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Abstract 


The  purpose  of  this  report  is  threefold.  First,  it  describes  how  Data  Envelopment 
Analysis  (DEA)  may  be  applied  to  assist  the  Manufacturing  Extension  Partnership  (MEP) 
Program  and  its  network  of  Manufacturing  Extension  Centers  (MECs)  in  assessing 
performance.  DEA  is  an  approach  to  performance  assessment  which  uses  linear 
programming  and  principles  of  frontier  analysis.  DEA  provides  an  integrated  method  for 
performance  assessment  which  computes  a single  measure  of  performance  based  on 
multiple  outputs  and  inputs.  To  accomplish  the  first  purpose,  the  theory  and  methodology 
of  DEA  are  related  to  MEP  operating  data  focused  on  the  twin  informational  needs  of 
continuous  improvement  and  program  accountability.  These  operating  data  include  both 
inputs— staff  and  financial  resources— and  outputs— types  and  levels  of  services  rendered. 
The  second  purpose  is  to  apply  data  from  the  current  NIST/MEP  Management 
Information  Reporting  System  to  illustrate  ways  in  which  DEA  can  help  MEP 
headquarters  and  its  network  of  MECs  to  measure  their  performance.  Operating  data 
from  51  MECs  are  analyzed  via  a series  of  performance  assessment  models;  each  model 
is  a unique  combination  of  inputs  and  outputs.  The  third  purpose  is  to  stimulate 
feedback  and  discussion  among  those  engaged  in  MEP  performance  assessment.  DEA 
provides  a method  which  MEP  and  its  network  of  MECs  can  use  jointly  to  make  sense  of 
the  multi-dimensional  nature  of  MEC  performance  data.  With  this  report  MEP  is  in  a 
position  to  initiate  further  uses  of  DEA  which  will  prove  beneficial  to  both  the  MECs  and 
the  national  program. 


Key  Words 

competitiveness:  data  envelopment  analysis:  economic  analysis:  economic  impacts: 
extension  services;  linear  programming:  manufacturing;  performance  assessment; 
program  evaluation. 


11 


Acknowledgments 

The  author  wishes  to  thank  all  those  who  contributed  so  many  excellent  ideas  and 
suggestions  which  he  has  attempted  to  incorporate  into  this  report.  The  author  wishes  to 
thank  Messrs.  John  Redman  and  Phillip  Blackerby  of  the  Manufacturing  Extension 
Partnership  (MEP)  Program  for  their  technical  guidance,  suggestions,  and  support. 

Special  appreciation  is  extended  to  Drs.  Stephen  F.  Weber  and  Harold  E.  Marshall  of  the 
Office  of  Applied  Economics  at  NIST's  Building  and  Fire  Research  Laboratory  (BFRL) 
for  the  thoroughness  of  their  reviews  and  for  their  many  insights  and  to  Ms.  Sandy  Kelley 
for  her  assistance  in  preparing  the  manuscript  for  review  and  publication.  Special 
appreciation  is  also  extended  to  Dr.  Gregory  A.  Norris,  formerly  with  BFRL's  Office  of 
Applied  Economics,  for  his  early  illustrations  of  how  data  envelopment  analysis  (DEA) 
could  be  applied  to  the  MEP.  Special  appreciation  is  also  extended  to  Ms.  Julia  D. 
Rhoten  of  Virginia  Polytechnic  Institute  and  State  University,  formerly  with  BFRL's 
Office  of  Applied  Economics,  who  helped  immensely  with  the  analysis  effort  during  her 
summer  and  winter  internships.  Thanks  are  due  to  Drs.  Saul  I.  Gass  and  Christoph 
Witzgall  of  NIST's  Information  Technology  Laboratory  and  Dr.  Mark  A.  Ehlen  of 
BFRL's  Office  of  Applied  Economics  for  their  comments  on  an  earlier  draft  of  this 
report. 


IV 


Table  of  Contents 


Abstract i 

Key  Words i 

Acknowledgments iii 

1 . Introduction 1 

1.1.  Background 1 

1.2.  Purpose 1 

1 .3 . Scope  and  Approach 2 

2.  Data  Envelopment  Analysis  (DEA):  A Summary  of  Key  Concepts 3 

2.1.  DEA  and  the  Manufacturing  Extension  Partnership  (MEP)  Program 3 

2.1.1.  Brief  Description  of  the  MEP  Program 4 

2.1.2.  Performance  Assessment 4 

2.1.3.  Continuous  Improvement 6 

2.2.  The  Performance  Frontier 8 

2.3.  Calculating  Efficiencies 9 

2.4.  The  Identification  of  Reference  Centers 10 

3.  Assessing  Manufacturing  Extension  Center  (MEC)  Performance:  Data,  Variables, 

and  Models 1 1 

3.1.  Brief  Description  of  the  MEC  Operating  Data  Sets 12 

3.2.  Selection  of  Variables  for  Analysis 13 

3.3.  Selection  of  Models  for  Analysis 14 

4.  Assessing  MEC  Performance:  Results 21 

4. 1 . Efficiency  Ranges  and  Reference  Centers  for  Each  MEC 21 

4.2.  Cross-Model  Comparisons 26 

4.3.  Interpretation  of  Selected  Results 35 

5.  Summary,  Conclusions,  and  Suggestions  for  Further  Research 39 

5.1.  Summary 39 

5.2.  Conclusions 40 

5.3.  Suggestions  for  Further  Research 41 

5.3.1.  Selection  of  Variables  and  Models:  Towards  a “Standard1'  Set  of  Variables 

and  Models 41 

5.3.2.  Feedback  Reports  for  Individual  Centers 42 

5.3.3.  Process  for  Benchmarking  Among  Centers 42 

5.3.4.  Process  for  Measuring  the  Performance  of  the  MEP  Program 42 

Appendix  A.  A Graphical  Approach  to  Performance  Assessment 43 


V 


Appendix  B.  Mathematical  Formulation 
References 


49 

51 


List  of  Figures 

Figure  2.  1 MEC  Operations  are  Multi-Dimensional . 5 

Figure  2.  2 MECs  and  the  Performance  Frontier . 7 

Figure  2.  3 Pushing  the  Performance  Frontier  Outwards 8 

Figure  2.  4 Efficiency  and  the  Performance  Frontier .....10 

Figure  4.  1 Measuring  the  Effects  of  Changing  the  Outputs  and  Inputs  on  the 

Production  Frontier 36 

Figure  A.l  Efficiency  Calculations  for  the  C001  MEC 44 


List  of  Tables 


Table  3.  1 List  of  Variables  for  "Standard"  Analyses  with  DEA 15 

Table  3.  2 List  of  Special  Variables  for  "Exploratory"  Analyses  with  DEA... 16 

Table  3.  3 Models  Analyzed  with  DEA:  Based  on  "Standard"  Output  Metrics  and 

Staff  Input  Metrics 18 

Table  3.  4 Models  Analyzed  with  DEA:  Based  on  "Standard"  Output  Metrics  and 

Financial  (Cash)  Input  Metrics 19 

Table  3.  5 Models  Analyzed  with  DEA:  Based  on  "Standard"  Output  Metrics  and 

"Indexed"  Financial  (Cash)  Input  Metrics. 20 

Table  4.  1 MEC  Efficiency  Ranges  and  Reference  Centers:  Based  on  "Standard" 

Output  Metrics  and  Staff  Input  Metrics  (Staff  Model  16) 23 

Table  4.  2 MEC  Efficiency  Ranges  and  Reference  Centers:  Based  on  "Standard" 

Output  Metrics  and  Financial  Input  Metrics  (Financial  Model  16) ....24 

Table  4.  3 MEC  Efficiency  Ranges  and  Reference  Centers:  Based  on  "Standard" 

Output  Metrics  and  "Indexed"  Financial  Input  Metrics  ("Indexed"  Financial 
Model  16) 


25 


"1 

pj 


Table  4.  4 Summary  of  Key  Results  for  Models  Analyzed  with  DEA:  Based  on 

"Standard"  Output  Metric  and  Staff  Input  Metrics 29 

Table  4.  5 Summary  of  Key  Results  for  Models  Analyzed  with  DEA:  Based  on 

"Standard"  Output  Metrics  and  Financial  Input  Metrics 30 

Table  4.  6 Summary  of  Key  Results  for  Models  Analyzed  with  DEA:  Based  on 

"Standard"  Output  Metrics  and  "Indexed"  Financial  Input  Metrics 31 

Table  A.l  Summary  of  Key  MEC  Operating  Data 43 


J 


j 

j 

j 


Vll 


1.  Introduction 

1.1.  Background 

The  pressures  of  competing  in  the  global  marketplace  are  affecting  nearly  every  U.S. 
business.  Now  more  than  ever.  U.S.  businesses  are  finding  that  they  must  continuously 
improve  their  products  and  services  if  they  are  to  survive  and  prosper.  These  competitive 
pressures  are  especially  high  for  more  than  370.000  U.S.  manufacturers  with  fewer  than 
500  employees.  These  small  and  medium-sized  manufacturers  account  for  about  95 
percent  of  all  U.S.  manufacturing  establishments  and  about  half  of  the  nation's 
manufacturing  capacity. 1 

NIST's  Manufacturing  Extension  Partnership  (MEP)  Program  is  a nationwide  network  of 
affiliated  Manufacturing  Extension  Centers  (MECs)  dedicated  to  helping  small  and 
medium-sized  manufacturers  improve  their  competitiveness  by  adopting  modem 
technologies  and  production  techniques.  Each  MEC  is  a partnership  typically  involving 
federal,  state,  and  local  governments;  industry;  educational  institutions;  and  other  sources 
of  expertise,  information,  and  funding  support.” 

Smaller  manufacturers  face  some  unique  barriers  that  hinder  their  ability  to  successfully 
modernize  their  operations.  The  MEP  links  these  manufacturers  with  information  and 
experienced  manufacturing  experts  to  help  them  improve  their  abilities  to  compete.  The 
MECs  serve  as  a conduit  for  these  manufacturers  to  in-house  manufacturing  experts,  as 
well  as.  private  consultants,  vendors,  universities,  federal  agencies,  and  other  sources  of 
technical  help. 

MEP  also  helps  foster  a more  unified  network  by  working  with  MECs  to  identify  and 
coordinate  the  services,  technology,  and  information  needed  at  a national  scale.  MEP  is 
developing  a uniform  system  to  help  MECs  evaluate  and  continuously  improve  the 
success  of  services  they  deliver.  This  report  is  intended  to  support  these  efforts. 

1.2.  Purpose 

The  purpose  of  this  report  is  threefold,  first,  it  describes  how  Data  Envelopment 
Analysis  (DEA)  may  be  applied  to  assist  the  Manufacturing  Extension  Partnership  (MEP) 
Program  and  its  network  of  Manufacturing  Extension  Centers  (MECs)  in  assessing 
performance.  Specifically,  the  report  provides  an  introduction  to  the  theory  and 
methodology  of  DEA.  For  a more  thorough  and  detailed  description  of  DEA.  see  the 


1 National  Institute  of  Standards  and  Technology  (NIST).  1995.  Technology’ for  Business.  NIST  Special 
Publication  875.  Gaithersburg,  MD:  National  Institute  of  Standards  and  Technology. 

“ Readers  interested  in  learning  more  about  the  MEP  Program  are  encouraged  to  browse  the  Manufacturin 
Extension  Partnership  Source,  an  electronic  resource  for  MEP-affi bated  MECs  and  the  manufacturing 
community  nationwide  (the  URL  Address  for  the  electronic  resource  is:  http://www.mep.nist.gov/). 
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excellent  book  by  Norman  and  Stoker.1  For  a concise  theoretical  discussion  of  DEA,  see 
the  review  article  by  Boussofiane  et  al .* * 4  To  accomplish  the  first  purpose,  the  theory  and 
methodology  of  DEA  are  related  to  MEP  operating  data  focused  on  the  twin 
informational  needs  of  continuous  improvement  and  program  accountability.  These 
operating  data  include  both  inputs— staff  and  financial  resources— and  outputs— types  and 
levels  of  services  rendered. 

Second,  the  report  uses  data  from  the  NIST/MEP  Management  Information  Reporting 
System  to  illustrate  ways  in  which  DEA  can  help  MEP  headquarters  and  its  network  of 
MECs  to  measure  their  performance.  To  accomplish  the  second  purpose,  operating  data 
from  51  MECs  are  analyzed  via  a series  of  performance  assessment  models.  Each  model 
is  a unique  combination  of  inputs  and  outputs  (i.e.,  variables).  The  models  illustrate  how 
changing  the  set  of  inputs  and/or  outputs— adding  a new  variable  or  disaggregating  an 
existing  one— affects  a particular  MEC's  performance. 

Third,  this  report  is  intended  to  stimulate  feedback  and  discussion  among  those  engaged 
in  MEP  performance  assessment.  DEA  provides  a method  which  MEP  and  its  network  of 
MECs  can  use  jointly  to  make  sense  of  the  multi-dimensional  nature  of  MEC 
performance  data.  While  DEA  is  only  one  of  several  MEP  approaches  to  performance 
assessment,  it  is  a particularly  flexible  and  interactive  one.  With  this  report  MEP  is  in  a 
position  to  initiate  further  uses  of  DEA  which  will  prove  beneficial  to  both  the  MECs  and 
the  national  program. 

1.3.  Scope  and  Approach 

This  report  has  four  chapters  and  two  appendices  in  addition  to  the  Introduction.  Chapter 

2 introduces  DEA,  describes  how  DEA  may  be  applied  to  the  MEP.  and  demonstrates 
ways  in  which  DEA  may  be  employed  to  drive  continuous  improvement  efforts.  Chapter 

3 describes  the  data  set  evaluated  with  DEA.  Special  attention  is  placed  on  the  variables 
analyzed  with  DEA  and  how  these  variables  are  used  to  specify  models  for  the  data  set. 
Chapter  4 contains  the  results  of  the  analysis.  Detailed  results  are  presented  in  several 
different  ways  in  order  to  generate  a deeper  understanding  of  how  DEA  may  be  used  to 
drive  performance  improvement  within  a individual  MEC  and  for  the  overall  MEP 
Program.  Chapter  5 concludes  the  report  with  a summary,  conclusions,  and  suggestions 
for  further  research.  The  basic  concepts  behind  DEA  described  in  Chapter  2 are  covered 
in  greater  detail  in  Appendix  A and  Appendix  B.  Appendix  A presents  a graphical 
approach  to  performance  assessment.  Appendix  B includes  a mathematical  formulation 
of  the  DEA  model  structure  used  throughout  this  report. 


' Norman,  Michael,  and  Stoker,  Barry.  1991.  Data  Envelopment  Analysis:  The  Assessment  of  Performance. 

New  York:  John  Wiley  and  Sons. 

4 Boussofiane,  A.,  Dyson,  R.G.  and  Thanassoulis,  E.  1991.  “Applied  Data  Envelopment  Analysis," 
European  Journal  of  Operational  Research , Vol.  52,  pp.  1-15. 
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2.  Data  Envelopment  Analysis  (DEA):  A Summary  of  Key  Concepts 


DEA  is  a mathematical  technique  that  provides  an  objective  assessment  of  the  operating 
efficiency  of  each  of  a number  of  similar  organizational  units,  relative  to  each  other. 
Operating  efficiencies  are  derived  through  reference  to  a small  set  of  units  which 
collectively  define  a performance  frontier.  Units  on  the  frontier  are  said  to  be  efficient  in 
the  allocation  of  their  inputs  and  the  production  of  their  outputs  while  units  not  on  the 
frontier  are  inefficient.  It  is  important  to  note  that  units  on  the  frontier  exhibit  actual 
achieved  performance  rather  than  theoretically  optimal  performance.  Consequently,  units 
on  the  frontier  reflect  actual  patterns  of  resource  allocation  and  production  rather  than 
theoretically  optimal  patterns. 

Apart  from  the  measure  of  relative  efficiency  of  each  unit,  DEA  also  yields  other 
information  which  proves  useful  in  gaining  a better  insight  into  the  performance  of  each 
unit  and  in  guiding  units  to  improve  their  performance.  Specifically,  DEA  identifies 
efficient  peer  units  for  every  inefficient  unit.  These  efficient  units  can  prove  useful  for 
identifying  efficient  operating  practices  which  can  be  disseminated  to  all  units  so  that 
they  may  improve  their  performance. 

Furthermore,  DEA  permits  differing  combinations  of  input  and  output  variables  to  be 
analyzed  and  evaluated.  Each  combination  of  variables  is  a specific  model.  Cross-model 
comparisons  can  then  be  used  to  evaluate  the  implications  of  adding  a new  variable  or 
disaggregating  an  existing  one.  Cross-model  comparisons  serve  to  highlight  exemplary 
practices  for  each  efficient  peer  unit,  producing  “paths  to  improved  performance"  for  less 
efficient  units. 

2.1.  DEA  and  the  Manufacturing  Extension  Partnership  (MEP)  Program 

There  are  a number  of  reasons  why  DEA  is  ideally  suited  for  use  by  the  MEP  Program. 
First,  MEP  operates  a national  network  of  Manufacturing  Extension  Centers  (MECs). 
Each  MEC  employs  similar  types  of  resources  as  inputs  and  provides  similar  types  of 
services  as  outputs.  Thus  the  “similar  organizational  units"  referred  to  above  are  MECs. 
Second,  DEA  enables  MEP  to  take  a long-term  view  of  performance  assessment  rather 
than  a single  snapshot  of  program  performance.  For  example,  a particular  model,  or  set 
of  models,  can  be  evaluated  with  successive  data  sets  to  measure  the  effects  of  adding 
“new"  MECs  and/or  the  effects  of  inter-temporal  differences  in  the  values  of  key 
variables  on  a particular  MEC's  performance.  Third.  MEP  has  established  a 
management  information  reporting  system  which  allows  a tight  coupling  between 
information  collected  for  programmatic  reasons  and  information  used  for  performance 
assessment  via  DEA.  Finally,  DEA  is  a tool  not  only  for  performance  assessment  but  for 
continuous  improvement  as  well.  Thus  DEA  may  be  used  both  for  improving  the 
performance  of  individual  MECs  and  of  the  overall  MEP  Program. 


To  better  understand  how  these  pieces  fit  together,  it  is  instructive  to  learn  more  about  the 
MEP  Program.  The  next  three  subsections  provide  background  information  on  the  MEP 
Program  with  particular  emphasis  on  the  role  of  DEA  as  a performance  assessment  and 
continuous  improvement  tool. 

2.1.1.  Brief  Description  of  the  MEP  Program 

The  NIST/MEP  Program  is  a national  network  of  services  to  assist  smaller  manufacturers 
in  becoming  globally  competitive.  MEP  combines  federal  support  with  state  and  local 
organizations  to  deliver  services  which  address  the  critical  and  often  unique  needs  of 
smaller  manufacturers.  In  1988,  Congress  directed  NIST  to  begin  helping  the  nation's 
smaller  manufacturers  adopt  and  apply  performance-improving  technologies  as  needed  to 
meet  intensifying  domestic  and  global  competition  in  manufacturing.  NIST  was  selected 
for  this  role  because  of  its  expertise  in  manufacturing  engineering  and  its  long-standing 
tradition  of  productive  partnerships  forged  with  public  and  private  organizations  at  the 
national,  state,  and  local  levels.  To  carry  out  this  role,  the  MEP  conducts  a variety  of 
regional,  national,  and  program  development  activities.  Regionally,  MEP  works  with  the 
states  or  local  organizations  to  establish  Manufacturing  Extension  Centers  (MECs)  or 
expand  existing  services  that  assist  smaller  manufacturers. 

MECs  are  designed  to  help  link  sources  of  improved  manufacturing  technology  and  the 
small  and  mid-sized  companies  that  need  it.  MEC  staff  work  with  individual  companies 
or  with  groups  of  companies  organized  around  common  needs,  industries,  or 
technologies.  While  each  MEC  tailors  its  services  to  meet  the  needs  dictated  by  its 
location  and  manufacturing  client  base,  some  common  services  are  offered  by  most 
MECs.  Broadly,  these  include  helping  manufacturers  assess  their  current  technology  and 
business  needs,  define  avenues  of  change,  and  implement  improvements. 

MEP's  activities  also  include  helping  foster  a more  unified  network  by  working  with 
MECs  to  identify  and  coordinate  the  services,  technology,  and  information  needed  at  the 
national  scale.  MEP  is  developing  a uniform  system  to  help  MECs  evaluate  and 
continuously  improve  the  success  of  services  they  deliver. 

2.1.2.  Performance  Assessment 

DEA  provides  a method  which  MEP  and  its  network  of  MECs  can  use  jointly  to  make 
sense  of  the  multi-dimensional  nature  of  MEC  performance  data.  DEA  can  augment, 
complement,  and  in  important  ways  improve  upon  an  MEC  assessment  that  uses 
individual  "performance  ratios,”  such  as  the  number  of  client  manufacturing 
establishments  (CMEs)  served  per  full-time  equivalent  employee  (FTE).  Performance 
assessment  via  DEA  can  be  conducted  in  a way  which  takes  into  account  the  influence  of 
special  circumstances,  such  as  the  number  of  CMEs  in  the  target  population,  which  may 
affect  performance  but  are  beyond  the  control  of  individual  MECs.  Consequently,  DEA 
allows  MECs  to  compare  their  own  performance  with  that  of  their  peers,  while  taking 
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diversity  of  MEC  outputs,  inputs,  and  special  circumstances  into  account.  While  DEA  is 
only  one  of  several  MEP  approaches  to  performance  assessment,  it  is  a particularly 
flexible  one. 

MECs  employ  a variety  of  resources  in  order  to  achieve  multiple  desired  outcomes  in 
client  firms  and  in  their  regional  economies.  As  seen  in  Figure  2.1,  MEC  resources 
(inputs)  and  MEC  outcomes  (outputs)  are  both  multi-dimensional.  Figure  2.1  shows  two 
broad  categories  of  inputs— staff  and  financial  resources.  The  MEC  must  deploy  its  staff 
and  financial  resources  on  a number  of ‘‘fronts''  to  achieve  its  objectives  (e.g.,  desired 
outcomes).  Three  types  of  outputs  are  shown  in  the  figure  to  capture  the  multi- 
dimensional nature  of  a typical  MEC’s  set  of  desired  outcomes.  Although  Figure  2.1  is  a 
simplification  of  how  an  MEC  operates,  it  provides  the  basic  elements  for  a meaningful 
evaluation  of  MEC  performance.  Consequently,  Figure  2.1  helps  us  to  understand  why 
any  attempt  to  assess  MEC  performance,  to  identify  peak  performers,  to  highlight 
approaches  which  are  achieving  the  greatest  success,  and  to  identify  improvement 
opportunities  for  each  MEC  must  take  account  of  the  multi -dimensional  nature  of  the 
MEC's  resources  and  outcomes. 


Inputs 


CMEs 


Outputs 


Figure  2.  1 MEC  Operations  are  Multi-Dimensional 

The  simplest  approach  to  evaluating  multi-dimensional  performance  is  to  calculate  a set 
of  performance  ratios  containing  a ratio  for  every  possible  pair  of  an  output  and  an  input 
(e.g.,  CMEs  served  per  FTE,  activities  > 8 hours  per  FTE,  etc.).  DEA  provides  an 
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integrated  method  for  performance  assessment  which  fulfills  the  need  to  address  multiple 
inputs  and  outputs  simultaneously.  Thus,  DEA  produces  a single  measure  of  performance 
by  taking  into  account  all  of  the  multiple  outputs  and  inputs.  In  contrast,  the  approach  of 
performance  ratios  would  lead  to  a bewildering  set  of  independent  and  incommensurate 
performance  ratios. 

MECs  operate  under  a wide  variety  of  circumstances.  Some  MECs  have  been  in 
existence  for  many  years,  while  others  are  newly  established.  Some  are  located  in  large 
and  dense  urban  areas,  while  others  are  situated  in  more  sprawling,  less  populated  areas. 
The  regional  economies  served  by  the  MECs  differ  in  ways  which  may  affect  measures  of 
MEC  performance.  For  example,  there  may  be  significant  variation  in  average 
manufacturing  firm  size,  dominant  manufacturing  industry  types,  costs  faced  by  regional 
manufacturers,  and  trends  in  the  overall  economy.  Any  attempt  to  comprehensively 
assess  MEC  performance  must  be  able  to  take  into  account  the  possible  influence  of  all 
these  factors.  DEA  fully  meets  this  challenging  requirement. 

2.1.3.  Continuous  Improvement 

DEA  provides  a framework  for  implementing  continuous  improvement  processes  both  for 
individual  MECs  and  for  the  overall  MEP  Program.  Two  key  concepts  are  employed  to 
promote  individual  MEC  performance:  (1 ) the  performance  frontier,  a line  representing 
the  maximum  attained  level  of  MEC  performance;  and  (2)  reference  centers,  those  MECs 
which  are  on  the  performance  frontier  and  are  most  similar  to  each  non-frontier  MEC. 
Both  concepts  are  illustrated  graphically  in  Figure  2.2.  Figure  2.2  represents  the  simplest 
case  involving  two  outputs  and  a single  input.  DEA  produces  a performance  frontier 
established  by  a small  but  well-defined  set  of  MECs.  These  frontier  MECs  demonstrate 
actual  achieved  peak  levels  of  performance.  Figure  2.2  contains  two  such  MECs— 
Frontier  Center  1 and  Frontier  Center  2.  The  frontier  MECs  provide  the  basis  for 
measuring  the  performance  of  “less  efficient"  non-frontier  MECs.  One  non-frontier 
MEC,  Non-Frontier  Center  A,  is  shown  in  Figure  2.2.  The  frontier  MECs  also  serve  as 
“reference  centers"  for  the  non-frontier  MECs.  In  the  case  illustrated  by  Figure  2.2, 
Frontier  Centers  1 and  2 are  the  reference  centers  for  Non-Frontier  Center  A. 

As  frontier  and  non-frontier  MECs  engage  in  benchmarking  each  others'  performance 
metrics,  improvements  or  new  approaches  to  service  delivery  will  be  employed.  As 
different  approaches  are  tried  and  refined,  some  non-frontier  MECs  will  improve  their 
performance  faster  than  those  currently  on  the  frontier.  This  will  likely  result  in  changes 
in  the  set  of  MECs  currently  on  the  frontier.  Thus,  the  reference  center  concept  should 
improve  not  only  individual  non-frontier  MEC  performance  (i.e.,  to  get  to  the  frontier) 
but  drive  improvement  for  frontier  MECs  as  well  (i.e.,  to  stay  on  the  frontier). 

A framework,  based  on  a time  series  of  DEA  analyses,  can  be  established  to  measure 
performance  progress  for  the  overall  MEP  Program.  For  example,  collection  of  results 
may  be  established  through  a series  of  applications  of  a specific  performance  assessment 
model.  Since  each  performance  assessment  model  consists  of  a fixed  set  of  variables,  it 
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provides  the  basis  for  an  index  measure— a single  summary  measure— for  each  MEC.  An 
index  measure  enables  us  to  track  each  MEC's  efficiency  changes  over  time  (e.g., 
efficiency  changes  resulting  from  inter-temporal  changes  in  the  quantities  of  inputs  and 
outputs).  For  example,  are  certain  MECs  consistently  on  the  frontier  and  how  has  the 
position  of  a non-frontier  MEC  changed  relative  to  the  frontier?  Consequently,  a time 
series  of  DEA  analyses  may  be  used  to  establish  a set  of  “rolling  benchmarks"  for  the 
MEP  Program.  These  rolling  benchmarks  will  enable  MEP  headquarters  to  strengthen  its 
continuous  improvement  efforts  by  focusing  on  pushing  the  entire  performance  frontier 
outwards. 


Output? 

Input 


Figure  2.  2 MECs  and  the  Performance  Frontier 

Figure  2.3  illustrates  a movement  of  the  entire  performance  frontier  from  the  initial  time 
period— the  Old  Frontier— to  the  new7  time  period— the  New7  Frontier.  The  figure  traces 
efficiency  changes  for  three  MECs  as  dotted  lines.  Note  that  Non-Frontier  Center  A has 
improved  its  performance  at  a faster  rate  than  the  two  original  frontier  MECs. 

Changes  in  the  values  of  the  index  measures  for  those  MECs  on  the  new7  frontier,  provide 
a straight-forw  ard  w ay  of  measuring  if  the  frontier  is  being  pushed  outw  ard,  or  if  not. 
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where  it  is  lagging.  In  the  case  illustrated  by  Figure  2.3,  the  entire  performance  frontier 
has  been  pushed  outward.  This  indicates  an  improvement  in  the  national  MEP  Program. 

2.2.  The  Performance  Frontier 

DEA  is  used  to  assess  the  performance  of  a number  of  comparable  entities  which 
consume  one  or  more  inputs  in  the  process  of  producing  one  or  more  outputs.  DEA 
refers  to  these  comparable  entities  as  “decision-making  units,”  or  DMUs.  The  concept  of 
a decision  making  unit  is  generic.  Consequently,  we  use  the  specific  term  MEC  to 
designate  the  decision  making  unit  (i.e.,  DMU)  of  interest. 


Output2 

Input 


Figure  2.  3 Pushing  the  Performance  Frontier  Outwards 

DEA  provides  answers  to  two  types  of  questions  which  arise  when  assessing  the 
performance  of  MECs:  (1)  how  well  is  each  MEC  doing;  and  (2)  how  much  better  could 
each  MEC  do.  In  developing  answers  to  both  questions,  the  benchmark  against  which 
MEC  performance  is  assessed  by  DEA  is  demonstrated  peak  levels  of  performance.  In 
DEA  terminology,  the  locus  of  peak  performance  possibilities  is  represented  by  the 
performance  frontier  (e.g.,  see  Figures  2.2  and  2.3).  The  shape  and  location  of  the  DEA 
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performance  frontier  reflects  an  important  assumption.  Namely,  all  linear  combinations 
of  actual  MEC  performance  (i.e.,  frontier  MECs)  are  assumed  to  be  feasible. 

2.3.  Calculating  Efficiencies 

The  shape  and  location  of  the  frontier  also  allows  us  to  quantify  the  degree  of  under- 
performance (i.e.,  room  for  improvement)  for  each  MEC  which  is  not  100%  efficient  (i.e., 
on  the  frontier).  The  efficiency  of  each  MEC  is  calculated  as  the  radial  distance  (from  the 
origin)  to  its  plotted  performance,  divided  by  the  total  radial  distance  to  the  frontier  along 
the  path  through  its  plotted  performance.  DEA's  measure  of  efficiency  is  illustrated 
graphically  in  Figure  2.4.  Reference  to  the  figure  demonstrates  that  the  Ath  MEC  is 
compared  with  that  frontier  point  which  has  its  same  relative  proportions  of  inputs  and 
outputs.  Appendix  A provides  a detailed,  step-by-step  procedure  for  calculating  the 
efficiency  of  a non-frontier  MEC.  The  procedure  is  graphically  oriented  and  uses  data 
from  only  three  MECs. 

DEA  also  produces  a set  of  “peer  units"  or  “reference  centers"  associated  with  each  MEC 
not  located  on  the  frontier.  An  interior  (non-frontier)  MEC's  reference  centers  are  those 
MECs  on  the  frontier  which  are  most  similar  to  the  interior  MEC  in  terms  of  their  relative 
ratios  of  inputs  and  outputs.  The  frontier  point  with  which  each  interior  MEC  is 
compared  is  the  linear  combination  of  its  reference  centers  that  results  in  its  exact  same 
relative  ratios  of  inputs  and  outputs  (see  Figure  2.4). 

The  efficiency  of  each  MEC  on  the  frontier  is  defined  as  being  equal  to  1 .0  (i.e.,  100%). 
Additionally,  the  efficiency  of  each  MEC  not  on  the  frontier  is  calculated  as  the  ratio  of 
two  radial  distances  defined  by  its  plotted  performance  and  that  frontier  point  which  has 
its  same  relative  proportions  of  inputs  and  outputs.  It  can  be  shown  that  this  approach  to 
calculating  efficiencies  is  equivalent  to  calculating  efficiencies  as  the  ratio  of  the 
weighted  sum  of  the  MEC's  outputs  divided  by  the  weighted  sum  of  its  inputs.  The 
individual  weights  for  each  MEC  are  chosen  to  maximize  its  own  calculated  efficiency, 
where  each  MEC's  weights  must  satisfy  two  sets  of  constraints.  First,  the  weights  are  all 
constrained  to  be  non-negative.  Second,  the  weights  are  subject  to  the  restriction  that 
neither  its  own  efficiency  nor  any  other  MEC's  efficiency  using  these  same  weights 
exceed  1.0.  While  this  model  formulation  is  non-linear,  the  model  formulation  may  be 
transformed  into  a linear  programming  problem.  The  resultant  linear  programming 
problem  is  individually  solved  for  each  MEC. 

Consider  the  case  where  there  are  n MECs.  When  solving  the  linear  programming 
problem  for  the  Ath  MEC,  the  input  weights  are  selected  so  that  the  weighted  sum  of  the 
inputs  equals  1.0.  The  output  weights  are  selected  to  maximize  the  weighted  sum  of  the 
outputs,  and  thus  the  efficiency  of  the  Alh  MEC.  Thus,  weights  are  selected  which  show 
each  MEC  in  the  best  possible  light.  Does  this  mean  that  all  MECs  are  found  to  be  100% 
efficient?  No,  because  the  second  set  of  constraints  requires  that  no  other  MEC  be  found 
more  than  100%  efficient  using  those  same  input  and  output  weights.  Note  that  this 
linear  programming  problem  is  solved  for  the  Ath  MEC.  Thus,  n of  these  separate  linear 
programming  problems  must  be  solved  to  generate  the  full  set  of  efficiency  scores.  The 
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mathematical  derivation  of  the  non-linear  model  formulation  and  the  specification  of  the 
linear  programming  problem  are  provided  in  Appendix  B. 


Figure  2.  4 Efficiency  and  the  Performance  Frontier 
2.4.  The  Identification  of  Reference  Centers 

Note  that  the  MECs  which  form  the  Ath  MEC's  reference  centers  are  precisely  those 
frontier  MECs  which  are  most  similar  to  the  Ath  MEC  in  terms  of  their  relative  ratios  of 
inputs  and  outputs.  Linear  programming  produces  an  optimum  set  of  input  and  output 
weights  which  maximize  the  Ath  MEC’s  efficiency.  Both  the  weights  and  the  calculated 
efficiency  for  the  Ath  MEC  are  sensitive  to  the  shape  and  location  of  the  performance 
frontier.  Because  those  frontier  centers  which  affect  the  Ath  MEC's  weights  and 
calculated  efficiency  are  its  reference  centers  (see  Appendices  A and  B),  a procedure  for 
identifying  reference  centers  is  needed. 

The  identification  of  reference  centers  is  accomplished  through  a post-processing 
operation.  This  operation  examines  if  one  or  more  of  a series  of  constraints  is  binding 
(see  Appendix  B).  If  the  constraint  is  binding,  then  the  efficiency  of  the  Ath  MEC  would 
be  higher  if  that  constraint  were  relaxed.  Those  constraints  which  are  binding  for  the  Ath 
MEC  correspond  to  its  reference  centers. 
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3.  Assessing  Manufacturing  Extension  Center  (MEC)  Performance: 
Data,  Variables,  and  Models 

This  chapter  describes  the  data  upon  which  all  DEA  analyses  were  based,  the  variables 
which  were  employed  in  these  analyses,  and  the  specification  of  each  of  the  performance 
assessment  models  evaluated  with  DEA.  All  data  are  derived  either  from  various  MEP 
reports  or  are  from  the  U.S.  Bureau  of  the  Census.  The  MEP  reports  are  of  two  types:  ( 1 ) 
compiled  sets  of  Monthly  Reports;  and  (2)  the  Semi-Annual  Report.  Data  from  the  two 
MEP  reports  were  used  to  create  the  variables  used  in  a series  of  “standard"  analyses. 

Data  from  the  Census'  County  Business  Patterns'  were  used  to  create  the  variables  used 
in  a series  of ‘'exploratory"  analyses. 

To  better  understand  the  relationships  between  the  data,  the  two  types  of  variables,  and 
the  associated  performance  assessment  models,  background  information  on  the  MEP- 
related  data  is  first  presented.  This  background  information  includes  an  overview  of  the 
NIST/MEP  reporting  system.  A brief  description  of  the  operating  data  set  follows  (see 
Section  3.1).  Each  of  the  two  types  of  variables— “standard"  and  “exploratory1'— are  then 
described  (see  Section  3.2).  The  chapter  concludes  with  the  specification  of  each  of  the 
performance  assessment  models  evaluated  with  DEA  (see  Section  3.3). 

The  stakeholders  in  the  National  MEP  system  have  a diverse  set  of  informational  needs. 
Consequently,  the  NIST/MEP  Management  Information  Reporting  System  was  designed 
to  collect  operational  data  and  qualitative  information  to  meet  these  diverse  informational 
needs.  In  addition  to  these  needs,  the  NIST/MEP  reporting  system  was  designed  to  meet 
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the  informational  needs  related  to  continuous  improvement  and  program  accountability. 

Continuous  improvement  is  vital  to  the  long-term  success  of  the  MEP  Program.  The 
collection  and  analysis  of  information  can  provide  insight  into  the  factors  leading  to 
effective  and  efficient  provision  of  services,  which  lead  to  continuous  improvement.  The 
reporting  system  standardizes  definitions  of  important  concepts  which  permit  the 
aggregation  and  statistical  analysis  of  data  that  can  build  the  knowledge  to  facilitate 
improvements.  DEA  provides  a framework  for  implementing  continuous  improvement 
processes  both  for  individual  MECs  and  the  overall  MEP  Program. 

NIST/MEP  is  charged  with  insuring  the  efficacy  of  taxpayer  investment  in  the  National 
system.  The  reporting  system  allows  NIST/MEP  to  monitor  MEC  performance  and  to 
ensure  that  the  MEP  Program  is  meeting  its  goal  of  “strengthening  the  global 
competitiveness  of  smaller  U.S.  manufacturers.”7  DEA  provides  a method  which  MEP 
and  its  network  of  MECs  can  use  jointly  to  make  sense  of  the  multi-dimensional  nature  of 
MEC  performance  data. 


' U.S.  Bureau  of  the  Census.  1994.  County  Business  Patterns.  Washington,  DC:  U.S.  Bureau  of  the  Census. 

For  a detailed  description  of  the  NIST/MEP  reporting  system  see.  National  Institute  of  Standards  and 
Technology  (NIST).  1996.  NIST/MEP  Management  Information  Reporting  Guidelines.  Gaithersburg,  MD: 
National  Institute  of  Standards  and  Technology. 

Ibid.,  p.1-2. 


Two  key  components  of  the  reporting  system  are  the  Monthly  Report  and  the  Semi- 
Annual  Report.  The  Monthly  and  Semi-Annual  Reports  provided  the  basis  for  all  of  the 
metrics  used  in  the  “standard"  analyses  of  MEC  operating  data. 

The  Monthly  Report  is  a set  of  two  documents  that  convey  information  regarding  an 
MEC’s  finances,  activities,  and  progress  towards  its  plans  as  outlined  in  its  yearly 
Operating  Plan.  The  first  document  is  a one-page  template  primarily  designed  to  assist 
MEP  Regional  Managers  in  assessing  the  MEC's  recent  expenditures  and  activity  levels 
by  tracking  MEC  performance  over  time.  The  first  document  also  serves  to  update  the 
central  NIST/MEP  database.  The  second  document  is  a brief  free-form  text  document 
that  provides  a qualitative  perspective  on  an  MEC's  progress  toward  its  overall  goals  as 
outlined  in  its  Operating  Plan. 

The  Semi-Annual  Report  is  made  up  of  several  reporting  documents,  some  of  which 
serve  to  update  the  central  NIST/MEP  database,  and  others  of  which  provide  information 
on  various  MEC  activities.  The  Semi-Annual  Report  Data  Page  is  a one-page  template 
that  captures  information  characterizing  the  cumulative  activities  of  the  MEC  over  time. 
Specifically,  it  is  designed  to  capture  the  total  number  of  activities  and  events  the  MEC 
has  completed  and  the  number  of  client  manufacturing  establishments  (CMEs)  with 
which  the  MEC  has  worked.  The  Semi-Annual  Report  Data  Page  is  a critical  component 
of  the  central  NIST/MEP  data  base.  Data  compiled  from  the  Semi-Annual  Report  Data 
Page  are  at  the  core  of  all  DEA  Analyses. 

3.1.  Brief  Description  of  the  MEC  Operating  Data  Set 

The  MEC  operating  data  set  analyzed  with  DEA  covers  the  last  six  months  of  1996.  This 
information  was  drawn  from  a compiled  set  of  Monthly  Reports  and  the  Semi-Annual 
Report  of  the  last  six  months  of  1996.  The  MEC  operating  data  set  used  in  this  report 
also  incorporated  supplementary  information  from  the  Census'  County  Business  Patterns. 
The  “original"  MEC  operating  data  set,  supplemented  with  information  from  the  County 
Business  Patterns , contained  approximately  1 1 0 variables  on  each  MEC.  The  variables 
in  the  operating  data  set  provided: 

• information  needed  to  identify  each  MEC  (i.e.,  tags), 

• values  of  inputs  (e.g.,  dollar-denominated,  financial-type  figures,  number  of  monthly 
reports  submitted,  staff  counts,  etc.),  and 

• the  value  of  outputs  (e.g.,  activities  over  8 hours  in  duration  completed  during  the 
reporting  period,  number  of  client  manufacturing  establishments  (CMEs)  served, 
percent  of  CMEs  in  the  target  population  served,  indexes  to  deflate  expenses,  etc.). 

The  MEC  operating  data  are  drawn  from  a compiled  set  of  Monthly  Reports  and  the 
Semi-Annual  Report  for  the  period  July  1,  1996  through  December  31,  1996.  Because 


target  population  data  were  based  on  information  from  the  1 994  County  Business 
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Patterns , it  w?as  necessary  to  exclude  from  the  analysis  eight  MECs  which  did  not  have 
estimated  values  for  their  target  population.  Furthermore,  two  MECs  were  combined  into 
a composite  MEC  because  only  state-wide  information  on  target  population  was  available 
for  their  state.  The  resultant  data  set  contains  51  MECs. 

In  addition  to  operating  data  from  the  Monthly  and  Semi-Annual  Reports,  the  data  set 
contains  information  compiled  from  the  County  Business  Patterns  on  the  costs  of 
engineering  and  management  consulting  services.  These  data  were  used  to  construct  a 
“consultant  cost  index"  and  a series  of  weighted  cost  metrics.  These  “exploratory" 
metrics  are  used  to  identify  MECs  operating  in  high  cost  areas  which  use  their  financial 
resources  very  efficiently. 

3.2.  Selection  of  Variables  for  Analysis 

The  key  variables  being  analyzed  via  DEA  are  summarized  in  Tables  3.1  and  3.2.  The 
tables  provide  the  variable  name,  its  definition,  and  its  type.  Wherever  possible,  the  same 
variable  name  as  in  the  “original"  MEC  operating  data  set  is  used.  Those  variables  which 
are  newf  (i.e.,  were  derived  by  Calculation  from  the  original  MEC  operating  data  set)  have 
a ”C_"  as  part  of  their  name  and  the  designation  as  “Calculated"  under  the  Type  column 
heading.  Variables  are  classified  into  five  types:  (1)  tag  (i.e.,  identifiers);  (2)  original 
input:  (3)  original  output;  (4)  calculated  input;  and  (5)  calculated  output. 

The  inputs  (i.e.,  original  and  calculated)  listed  in  Table  3.1  are  classified  into  three  basic 
types:  (1)  the  number  of  full-time  equivalents— FTEs-(i.e.,  staff  metrics),  (2)  cash 
expenses  (i.e.,  financial  metrics),  and  (3)  the  number  of  CMEs  in  the  target  population.  It 
is  important  to  note  that  each  of  the  input  totals— FTEMOAVE  (total  FTEs.  all  personnel  ) 
and  KC_TOTAL  (“revised"  total  expenses,  cash  $K)— can  be  decomposed  into  tw'o 
components.  The  components  of  FTEMOAVE  are:  ADMOAVE  (FTEs.  administrative 
personnel)  and  ENGMOAVE  (FTEs.  field  agents  plus  technical  specialists).  The 
components  of  KC  TOTAL  are:  KSTOTPER  (personnel  expenses,  cash.  $K)  and 
KCOPSC  (“revised"  operations  expenses,  cash.  $Kfy  This  approach  enables  us  to 
decompose  the  effects  of  each  input  (i.e.,  staff  and  financial)  into  its  component  effects. 

Table  3.1  contains  five  outputs  (i.e..  original  and  calculated).  One  of  these  outputs. 
CMEPERID  (CMEs  served  during  the  reporting  period),  can  be  decomposed  into  two 
components:  CMEFIRST  (CMEs  served  for  the  first  time)  and  CCMERP  (CMEs 
which  are  repeat  business).  Another  key  output,  ACT  S,  records  the  number  of  activities 
completed  greater  than  8 hours  in  duration.  This  output  provides  a measure  of  substantial 


Data  on  the  number  of  CMEs  in  the  target  population  were  derived  from  the  latest  available  (1994)  County 
Business  Patterns.  This  approach  was  chosen  because  it  resulted  in  a single,  consistent  definition  of  the 
target  population  across  all  MECs. 

‘ The  values  of  total  expenses  and  operations  expenses  have  been  "revised"  by  subtracting  out  the  value  of 
operations  expenses  for  third  party  providers,  THIRDPS.  This  adjustment  was  dictated  bv  data  anomalies 
associated  with  the  input  variable  THIRDPS. 


interaction  between  the  MEC  and  its  client  base.  Another  output,  C_PCTCME,  provides 
a measure  of  market  penetration  (i.e.,  what  percentage  of  the  target  population  of  CMEs 
was  served  this  period).  Thus,  the  key  outputs  analyzed  cover:  (1)  the  number  of  CMEs 
served,  including  the  breakout  between  new  and  repeat  business,  (2)  activities  which 
represent  substantial  interactions,  and  (3)  market  penetration. 

Table  3.2  records  the  seven  variables  used  in  a series  of  “exploratory’'  analyses.  The  first 
four  variables  are  used  to  develop  a consultant  cost  index.  The  consultant  cost  index  is 
used  to  deflate  expenses  (i.e.,  KC  TOTAL,  KSTOTPER.  and  KC  OPS  C),  adjusting 
each  MEC's  expenses  for  systematic  differences  in  costs  among  MECs.  This  “leveling  of 
the  playing  field"  allows  MECs  which  operate  in  a high-cost  area  but  make  efficient  use 
of  their  financial  resources  to  be  identified.  The  components  of  the  consultant  cost  index 
are  NDX  TOT,  NDX_PER.  and  NDX  OPS,  respectively.  Table  3.2  records  the  seven 
variables  used  in  a series  of  "exploratory'’  analyses. 

3.3.  Selection  of  Models  for  Analysis 

A series  of  performance  assessment  models  was  constructed  and  evaluated  with  DEA. 
Each  model  was  a unique  combination  of  inputs  and  outputs.  In  all,  48  models  were 
evaluated  with  DEA. 

Tables  3.3  through  3.5  list  the  models  analyzed  with  DEA.  The  models  specified  in 
Tables  3.3  through  3.5  make  use  of  the  same  basic  elements  shown  in  Figure  2.1 . Each 
set  of  models  is  built  around  one  of  the  key  inputs— either  staff  metrics  (FTEs)  or  financial 
metrics  (cash  expenses)— shown  in  Figure  2.1. 

Models  for  which  both  the  outputs  and  the  inputs  are  “standard"  (i.e..  the  values  for  all 
metrics  are  based  on  the  variables  listed  in  Table  3.1 ) are  referred  to  as  a “standard" 
model  type.  The  models  listed  in  Table  3.3  are  the  “standard"  models  for  staff  input 
metrics.  The  models  listed  in  Table  3.4  are  the  “standard"  models  for  financial  (cash) 
input  metrics. 

Models  for  which  the  values  of  the  input  metrics  are  based  on  the  variables  listed  in  Table 
3.2  are  referred  to  as  an  “exploratory"  model  type.  These  models  are  listed  in  Table  3.5. 

Tables  3.3  and  3.4  specify  the  “standard"  models  analyzed  with  DEA.  Each  table  has  16 
models.  The  name  of  each  of  the  16  models  is  given  under  the  column  heading.  Model. 
The  name  of  each  variable  which  is  an  output  is  listed  under  the  appropriate  column 
heading  (i.e.,  Output  No.  1 through  Output  No.  4).  The  name  of  each  variable  which  is  an 
input  is  listed  under  the  appropriate  column  heading  (i.e.,  Input  No.  1 and  Input  No.  2). 
The  structure  of  each  table  is  designed  to  reveal  the  differential  effect  of  adding  or 
disaggregating  an  output  and/or  disaggregating  the  input.  Note  that  the  outputs  are  the 
same  and  are  grouped  in  the  same  manner  in  both  tables,  so  that  the  effect  of  each  input 
metric  (staff  vs.  financial)  can  be  directly  observed  for  each  model.  The  inputs  are  based 
on  either  staff  metrics  (Table  3.3)  or  financial  metrics  (Table  3.4). 
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Table  3.  1 List  of  Variables  for  "Standard"  Analyses  with  DEA 


Variable  Name 

Definition 

Type 

CODE 

Center  Code 

Tag 

STOTPERS 

Personnel  Expenses,  Cash 

Original 

Input 

THIRDPS 

Operations  Expenses,  Third  Party  Service  Providers, 
Cash 

Original 

Input 

STOTOPSS 

Operations  Expenses,  Subtotal,  Cash 

Original 

Input 

C_OPS_C 

“Revised"  Operations  Expenses,  Cash  (Operations 
Expenses  (STOTOPSS)  Less  Third  Party  Expenses 
(THIRDPS)) 

Calculated 

Input 

CTOTAL 

“ Revised"  Total  Expenses,  Cash  (Personnel  Expenses 
(STOTPERS)  Plus  “Revised"  Operations  Expenses 
(COPSC) 

Calculated 

Input 

KSTOTPER 

Personnel  Expenses,  Cash,  $K 

Calculated 

Input 

KCOPSC 

“Revised”  Operations  Expenses,  Cash,  $K 

Calculated 

Input 

KCTOTAL 

“Revised"  Total  Expenses,  Cash,  $K 

Calculated 

Input 

ADMOAVE 

FTEs,  Administrative  Personnel 

Original 

Input 

ENGMOAVE 

FTEs,  Field  Agents  Plus  Technical  Specialists 

Original 

Input 

FTEMOAVE 

Total  FTEs,  All  Personnel 

Original 

Input 

CMETARGT 

CMEs  in  Target  Population,  Based  on  Data  Derived 
from  the  County  Business  Patterns 

Calculated 

Input 

ACT8 

Activities  > 8 Hours 

Original 

Output 

CMEPERID 

CMEs  Served  During  Reporting  Period 

Original 

Output 

CMEFIRST 

CMEs  Served  for  the  First  Time 

Original 

Output 

CCMERP 

CMEs  Served  which  are  Repeat  Business 
(CMEPERID  Less  CMEFIRST) 

Calculated 

Output 

CPCTCME 

Percent  of  CMEs  in  Target  Population  Served 
(((CMEPERID)/(CMETARGT))*  1 00) 

Calculated 

Output 
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Table  3.  2 List  of  Special  Variables  for  ’’Exploratory"  Analyses  with  DEA 


Variable  Name 

Definition 

Type 

ENGSAL 

Average  Salary  for  Engineering  Consulting 
Services  (SIC  Code  8711).  Based  on  Data  from 
County  Business  Patterns 

Calculated  Input 

MANSAL 

Average  Salary  for  Management  Consulting 
Services  (SIC  Code  8742),  Based  on  Data  from 
County  Business  Patterns 

Calculated  Input 

WGTCONS 

Weighted  Average  Salary  for  Consulting  Services 
for  this  MEC,  Based  on  Proportion  of  Total  FTEs 
Engaged  in  Engineering  or  Management  Services 
Times  the  Respective  Average  Salary 

Calculated  Input 

CONSNDX 

Index  Used  to  Deflate  Expenses,  Weighted 

Average  Salary  for  Consulting  Services  for  this 
MEC  (WGT  CONS)  Divided  by  the  Average 
Salary  for  Consulting  Services  Across  all  MECs 

Calculated  Input 

NDXPER 

Personnel  Expenses,  Cash,  $K,  Indexed  for 
Systematic  Cost  Differences  (KSTOTPER 

Divided  by  CONS  NDX) 

Calculated  Input 

NDX  OPS 

“Revised''  Operations  Expenses,  Cash,  $K, 

Indexed  for  Systematic  Cost  Differences 
(KC  OPS  C Divided  by  CONS  NDX) 

Calculated  Input 

NDXTOT 

“Revised"  Total  Expenses,  Cash  $K,  Indexed  for 
Systematic  Cost  Differences  (KC  TOTAL 

Divided  by  CONS_NDX) 

Calculated  Input 

16 


Table  3.3  records  the  16  “standard"  staff  models  analyzed  with  DEA.  The  first  set  of  four 
models  (Staff  Model  1 through  Staff  Model  4)  represents  all  possible  combinations  of 
two  outputs  or  a single  output  (CMEPERID)  disaggregated  into  its  two  component  parts. 
These  models  are  grouped  so  as  to  provide  the  basis  for  measuring  the  effects  of  either 
adding  another  output  or  disaggregating  FTEMOAVE  into  its  two  component  parts, 
ADMOAVE  (the  number  of  administrative  staff)  and  ENGMOAVE  (the  number  of  field 
staff).  The  second  set  of  four  models  (Staff  Model  5 through  Staff  Model  8)  maintains 
the  same  output  pattern  as  the  first  set  of  four  models  but  disaggregates  FTEMOAVE  into 
its  two  component  parts.  The  next  set  of  three  models  (Staff  Model  9 through  Staff 
Model  1 1 ) adds  a third  output  but  with  a single  input,  FTEMOAVE.  These  models 
enable  us  to  measure  the  effect  of  adding  a third  variable  or  of  disaggregating  the  number 
of  CMEs  served  into  its  two  component  parts.  The  next  set  of  three  models  (Staff  Model 
12  through  Staff  Model  14)  measures  the  effect  of  disaggregating  FTEMOAVE  into  its 
two  component  parts,  while  maintaining  three  outputs.  The  last  set  of  two  models 
measures  the  effect  of  adding  a fourth  output  with  a single  input  (Staff  Model  15)  and  of 
disaggregating  FTETMOAVE  into  its  two  component  parts  (Staff  Model  16). 

Table  3.4  records  the  16  “standard"  financial  models  analyzed  with  DEA.  The  first  set  of 
four  models  (Financial  Model  1 through  Financial  Model  4)  represents  all  possible 
combinations  of  two  outputs  or  a single  output  (CMEPERID)  disaggregated  into  its  two 
component  parts.  These  models  are  grouped  so  as  to  provide  the  basis  for  measuring  the 
effects  of  either  adding  another  output  or  disaggregating  KC  TOTAL  into  its  two 
component  parts,  KSTOTPER  (personnel  expenses)  and  KC_OPS_C  (operations 
expenses  less  third-party  expenses).  The  second  set  of  four  models  (Financial  Model  5 
through  Financial  Model  8)  maintains  the  same  output  pattern  as  the  first  set  of  four 
models  but  disaggregates  KC  TOTAL  into  its  two  component  parts.  The  next  set  of 
three  models  (Financial  Model  9 through  Financial  Model  1 1)  adds  a third  output  but 
with  a single  input.  KC_TOTAL.  These  models  enable  us  to  measure  the  effect  of  adding 
a third  variable  or  of  disaggregating  the  number  of  CMEs  served  into  its  two  component 
parts.  The  next  set  of  three  models  (Financial  Model  12  through  Financial  Model  14) 
measures  the  effect  of  disaggregating  KC  TOTAL  into  its  two  component  parts,  while 
maintaining  three  outputs.  The  last  set  of  two  models  measures  the  effect  of  adding  a 
fourth  output  with  a single  input  (Financial  Model  15)  and  of  disaggregating  KCTOTAL 
into  its  two  component  pails  (Financial  Model  16). 

Table  3.5  records  the  16  “indexed"  financial  models.  The  four  "standard"  output  metrics 
are  analyzed  via  an  indexed  set  of  financial  inputs.  Except  for  indexing,  the  models 
shown  in  Table  3.5  follow  the  same  structure  as  those  shown  in  Table  3.4.  Specifically, 
personnel  expenses,  “revised"  operations  expenses,  and  "revised"  total  expenses  are  all 
indexed  for  systematic  cost  differences.  These  “indexed”  financial  models  are  designed 
to  adjust  each  MEC's  expenses  for  systematic  differences  in  costs  among  MECs.  This 
approach  allows  MECs  which  operate  in  a high-cost  area  but  make  efficient  use  of  their 
financial  resources  to  be  identified. 
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Table  3.  3 Models  Analyzed  with  DEA:  Based  on  "Standard”  Output  Metrics  and  Staff  Input  Metrics 
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Table  3.  4 Models  Analyzed  with  DEA:  Based  on  "Standard"  Output  Metrics  and  Financial  (Cash)  Input  Metrics 
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Table  3.  5 Models  Analyzed  with  DEA:  Based  on  "Standard”  Output  Metrics  and  "Indexed"  Financial  (Cash)  Input 

Metrics 
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4. 


Assessing  MEC  Performance:  Results 


Tables  4.1  through  4.6  summarize  the  results  from  applying  DEA  to  each  of  the  models 
listed  in  Tables  3.3  through  3.5.  The  first  set  of  three  tables.  Tables  4.1  through  4.3. 
covers  the  disaggregated-output.  two-input  models  (e.g.,  Table  4.1  covers  Staff  Model  16 
from  Table  3.3,  Table  4.2  covers  financial  Model  16  from  Table  3.4.  and  Table  4.3  cover 
“Indexed"  Financial  Model  16  from  Table  3.5). 

The  second  set  of  three  tables  provides  results  for  each  set  of  models  specified  in  Tables 
3.3  through  3.5  (e.g..  Table  4.4  covers  the  16  staff  models  specified  in  Table  3.3.  Table 
4.5  covers  the  16  financial  models  specified  in  Table  3.4.  and  Table  4.6  covers  the  16 
“indexed"  financial  models  specified  in  Table  3.5). 

4.1.  Efficiency  Ranges  and  Reference  Centers  for  Each  MEC 

Tables  4.1  through  4.3  list  the  calculated  efficiency  ranges  and  reference  centers  for  each 
of  the  most  detailed  models  listed  in  Tables  3.3  through  3.5.  All  three  tables  have  the 
same  basic  format.  This  was  done  to  simplify  comparisons  across  model  types.  In  the 
first  column  of  Tables  4.1  through  4.3.  the  name  of  each  MEC  is  listed  beneath  the 
column  heading  Center  Name.  The  data  set  analyzed  via  DEA  contained  information 
from  51  MECs.  In  order  to  simplify  the  reporting  of  results,  all  MECs  were  assigned  a 
sequence  number.  The  sequence  numbers  range  from  MEC01  to  MEC  51.  The  MECs 
are  listed  sequentially.  The  second  column  provides  the  calculated  efficiency  range  for 
each  MEC.  Those  MECs  which  define  the  performance  frontier  are  shown  in  bold  face. 
In  order  to  assign  each  MEC  to  an  efficiency  range,  all  MECs  were  first  rank-ordered 
according  to  their  calculated  efficiency.  Based  on  this  rank  ordering,  three  statisically- 
based  ranges  are  used  to  characterize  MEC  performance.  These  ranges  are:  (1)  the  top  25 
percent,  designated  as  the  upper-quartile  range:  (2)  the  bottom  25  percent,  designated  as 
the  lower-quartile  range:  and  (3)  the  middle  50  percent,  designated  as  the  inter-quartile 
range.  MECs  in  the  upper-quartile  range  are  further  subdivided  into  frontier  and  non- 
frontier MECs.  The  entries  in  the  second  column  are  as  follows:  (1)  Frontier  (i.e.,  an 
MEC  in  the  upper-quartile  range  which  is  on  the  frontier);  (2)  Upper  (i.e.,  an  MEC  in  the 
upper-quartile  range  which  is  NOT  on  the  frontier);  (3)  Inter  (i.e.,  an  MEC  which  is  in  the 
inter-quartile  range);  and  (4)  Lower  (i.e..  an  MEC  which  is  in  the  lower-quartile  range). 
Because  there  are  51  MECs.  both  the  upper-  and  lower-quartile  ranges  are  assumed  to 
contain  13  MECs. 

The  next  four  columns  of  Tables  4.1  through  4.3  contain  the  identities  of  each  MEC's 
reference  centers.  Recall  that  the  reference  centers  are  those  frontier  MECs  which  are 
most  similar— in  terms  of  their  relative  shares  of  inputs  and  outputs— to  the  MEC  under 
analysis.  It  is  important  to  recognize  that  both  the  number  of  frontier  MECs  and  the 
identities  of  the  reference  centers  are  model  dependent.  This  is  because  each  model 
represents  a unique  combination  of  inputs  and  outputs.  Consequently,  some  models  will 
have  more  frontier  MECs— due  to  a specific  combination  of  inputs  and  outputs— and.  thus, 
have  more  opportunities  for  producing  reference  centers. 
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Table  4.1  lists  the  calculated  efficiency  ranges  and  reference  centers  for  each  of  the  51 
MECs  analyzed  via  Staff  Model  16.  The  eight  MECs  which  define  the  performance 
frontier  are  shown  in  bold  face.  The  eight  MECs  are:  (1)  MEC04;  (2)  MEC09;  (3) 
MEC18;  (4)  MEC27;  (5)  MEC36;  (6)  MEC39;  (7)  MEC40;  and  (8)  MEC44.  Note  that 
the  eight  MECs  which  define  the  performance  frontier  are  their  own  reference  centers. 

For  non-frontier  MECs,  the  number  of  reference  centers  ranges  from  one  to  three. 

Table  4.2  lists  the  calculated  efficiency  ranges  and  reference  centers  for  each  of  the  51 
MECs  analyzed  via  Financial  Model  16.  The  six  MECs  which  define  the  performance 
frontier  are  shown  in  bold  face.  The  six  MECs  are:  (1)  MEC03;  (2)  MEC04;  (3) 

MEC09;  (4)  MEC1 1;  (5)  MEC27;  and  (6)  MEC37.  Note  that  the  six  MECs  which  define 
the  performance  frontier  are  their  own  reference  centers.  For  non-frontier  MECs,  the 
number  of  reference  centers  ranges  from  two  to  four. 

Table  4.3  lists  the  calculated  efficiency  ranges  and  reference  centers  for  each  of  the  51 
MECs  analyzed  via  '‘Indexed”  Financial  Model  16.  In  “Indexed"  Financial  Model  16,  the 
four  “standard”  output  metrics  are  analyzed  via  an  indexed  set  of  financial  inputs. 
Specifically,  personnel  expenses,  “revised"  operations  expenses,  and  “revised"  total 
expenses  are  all  indexed  for  systematic  cost  differences.  This  approach  allows  MECs 
which  operate  in  a high-cost  area  but  make  efficient  use  of  their  financial  resources  to  be 
identified.  The  eight  MECs  which  define  the  performance  frontier  are  shown  in  bold 
face.  The  eight  MECs  are:  (1)  MEC03;  (2)  MEC04;  (3)  MEC09;  (4)  MEC1 1;  (5) 
MEC26;  (6)  MEC27;  (7)  MEC37;  and  (8)  MEC39.  Note  that  the  eight  MECs  which 
define  the  performance  frontier  are  their  own  reference  centers.  For  non-frontier  MECs, 
the  number  of  reference  centers  ranges  from  two  to  four. 

Comparisons  between  Table  4.1,  4.2,  and  4.3  show  that  frontier  MECs  in  one  model  are 
not  necessarily  frontier  MECs  in  another  model.  For  example,  MEC1 1 is  a frontier  MEC 
for  Financial  Model  16  but  is  not  a frontier  MEC  for  Staff  Model  16.  Three  MECs — 
MEC04,  MEC09,  and  MEC27— are  on  the  frontier  for  all  three  of  the  models.  Because 
these  MECs  are  efficiently  allocating  both  their  staff  and  their  financial  resources,  they 
merit  a close  examination.  It  is  important  to  note  that  these  three  MECs  exhibit 
efficiency  through  “diversification”  (i.e.,  efficiency  in  the  utilization  of  both  their  staff 
and  financial  resources).  Furthermore,  results  like  these— where  a particular  MEC  is  on 
the  frontier  for  several  different  models,  especially  models  with  different  types  of  inputs— 
provide  a framework  for  identifying  “best  practices.”  Identifying  best  practices  is 
extremely  important  for  the  MEP  Program,  because  best  practices  are  a key  component 
for  establishing  a meaningful  benchmarking  process  among  MECs. 
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Table  4.  1 MEC  Efficiency  Ranges  and  Reference  Centers:  Based  on  ’’Standard" 
Output  Metrics  and  Staff  Input  Metrics  (Staff  Model  16) 


Center  Name 

Quartile 

Reference  Centers 

No.  1 

No.  2 

No.  3 

No.  4 

MEC01 

Inter 

MEC04 

MEC44 

MEC02 

Inter 

MEC44 

MEC40 

MEC03 

Inter 

MEC40 

MEC36 

MEC  04 

Frontier 

MEC04 

MEC05 

Inter 

MEC  18 

MEC04 

MEC40 

MEC06 

Lower 

MEC44 

MEC40 

MEC09 

MEC07 

Lower 

MEC04 

MEC40 

MEC  18 

MEC08 

Inter 

MEC44 

MEC40 

MEC09 

Frontier 

MEC09 

MEC  10 

Lower 

MEC40 

MEC27 

MEC  11 

Upper 

MEC40 

MEC09 

MEC  12 

Inter 

MEC44 

MEC  13 

Upper 

MEC04 

MEC44 

MEC40 

MEC14 

Upper 

MEC04 

MEC44 

MEC  15 

Lower 

MEC  18 

MEC40 

MEC  16 

Inter 

MEC40 

MEC04 

MEC39 

MEC  17 

Inter 

MEC  18 

MEC18 

Frontier 

MEC  18 

MEC  19 

Inter 

MEC40 

MEC04 

MEC39 

MEC20 

Lower 

MEC40 

MEC09 

MEC21 

Inter 

MEC44 

MEC40 

MEC22 

Lower 

MEC40 

MEC  18 

MEC23 

Upper 

MEC04 

MEC44 

MEC40 

MEC24 

Inter 

MEC40 

MEC  18 

MEC25 

Inter 

MEC44 

MEC40 

MEC26 

Upper 

MEC04 

MEC44 

MEC40 

MEC27 

Frontier 

MEC27 

MEC28 

Inter 

MEC44 

MEC40 

MEC29 

Inter 

MEC40 

MEC30 

Lower 

MEC40 

MEC39 

MEC31 

Low'er 

MEC40 

MEC27 

MEC32 

Inter 

MEC44 

MEC40 

MEC33 

Inter 

MEC44 

MEC40 

MEC34 

Inter 

MEC44 

MEC09 

MEC40 

MEC35 

Inter 

MEC  18 

MEC40 

MEC36 

Frontier 

MEC36 

MEC37 

Lower 

MEC  18 

MEC40 

MEC38 

Inter 

MEC44 

MEC40 

MEC39 

Frontier 

MEC39 

MEC  40 

Frontier 

MEC40 

MEC41 

Lower 

MEC  18 

MEC40 

MEC42 

Lower 

MEC44 

MEC43 

Inter 

MEC40 

MEC04 

MEC44 

MEC44 

Frontier 

MEC  44 

MEC45 

Lower 

MEC44 

MEC40 

MEC46 

Lower 

MEC40 

MEC39 

MEC47 

Inter 

MEC40 

MEC44 

MEC48 

Inter 

MEC44 

MEC40 

MEC49 

Inter 

MEC44 

MEC50 

Inter 

MEC40 

MEC18 

MEC51 

Inter 

MEC04 

MEC40 

MEC39 

Table  4.  2 MEC  Efficiency  Ranges  and  Reference  Centers:  Based  on  "Standard” 
Output  Metrics  and  Financial  Input  Metrics  (Financial  Model  16) 


Center  Name 

Quartile 

Reference  Centers 

No.  1 

No.  2 

No.  3 

No.  4 

MEC01 

Inter 

MEC04 

MEC03 

MEC1 1 

MEC02 

Inter 

MEC  11 

MEC09 

MEC03 

Frontier 

MEC03 

MEC04 

Frontier 

MEC04 

MEC05 

Inter 

MEC04 

MEC37 

MEC03 

MEC06 

Lower 

MEC03 

MEC1 1 

MEC04 

MEC07 

Lower 

MEC04 

MEC37 

MEC  11 

MEC09 

MEC08 

Inter 

MEC  11 

MEC03 

MEC09 

Frontier 

MEC  09 

MEC  10 

Lower 

MEC  11 

MEC37 

MEC09 

MEC  11 

Frontier 

MEC  11 

MEC12 

Inter 

MEC  11 

MEC09 

MEC  13 

Inter 

MEC04 

MEC  11 

MEC09 

MEC37 

MEC  14 

Inter 

MEC04 

MEC  11 

MEC09 

MEC  15 

Lower 

MEC37 

MEC03 

MEC04 

MEC  16 

Inter 

MEC1 1 

MEC37 

MEC09 

MEC04 

MEC  17 

Inter 

MEC03 

MEC37 

MEC04 

MEC  18 

Upper 

MEC04 

MEC37 

MEC09 

MEC  19 

Upper 

MEC  11 

MEC03 

MEC37 

MEC04 

MEC20 

Lower 

MEC37 

MEC1 1 

MEC03 

MEC04 

MEC21 

Inter 

MEC1 1 

MEC03 

MEC37 

MEC22 

Lower 

MEC03 

MEC37 

MEC  11 

MEC23 

Upper 

MEC04 

MEC1 1 

MEC09 

MEC24 

Lower 

MEC37 

MEC04 

MEC1 1 

MEC09 

MEC25 

Upper 

MEC  11 

MEC37 

MEC04 

MEC09 

MEC26 

Inter 

MEC04 

MEC03 

MEC  11 

MEC27 

Frontier 

MEC27 

MEC28 

Inter 

MEC1 1 

MEC03 

MEC04 

MEC29 

Inter 

MEC1 1 

MEC03 

MEC37 

MEC30 

Inter 

MEC1 1 

MEC37 

MEC03 

MEC31 

Lower 

MEC1 1 

MEC03 

MEC32 

Inter 

MEC1 1 

MEC04 

MEC03 

MEC33 

Upper 

MEC  11 

MEC03 

MEC37 

MEC34 

Inter 

MEC1 1 

MEC03 

MEC37 

MEC35 

Inter 

MEC03 

MEC37 

MEC36 

Inter 

MEC37 

MEC09 

MEC37 

Frontier 

MEC37 

MEC38 

Inter 

MEC04 

MEC37 

MEC1 1 

MEC09 

MEC39 

Upper 

MEC27 

MEC09 

MEC1 1 

MEC40 

Upper 

MEC03 

MEC1 1 

MEC37 

MEC41 

Lower 

MEC37 

MEC04 

MEC03 

MEC42 

Lower 

MEC03 

MEC1 1 

MEC43 

Inter 

MEC04 

MEC  11 

MEC09 

MEC44 

Inter 

MEC  11 

MEC04 

MEC09 

MEC37 

MEC45 

Lower 

MEC1 1 

MEC03 

MEC37 

MEC46 

Lower 

MEC11 

MEC03 

MEC37 

MEC47 

Lower 

MEC  11 

MEC04 

MEC37 

MEC09 

MEC48 

Inter 

MEC1 1 

MEC03 

MEC37 

MEC49 

Inter 

MEC  11 

MEC09 

MEC50 

Inter 

MEC37 

MEC03 

MEC  11 

MEC04 

MEC51 

Inter 

MEC04 

MEC1 1 

MEC09 
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Table  4.  3 MEC  Efficiency  Ranges  and  Reference  Centers:  Based  on  "Standard" 
Output  Metrics  and  "Indexed"  Financial  Input  Metrics  ("Indexed"  Financial 
Model  16) 


Center  Name 

Quartile 

Reference  Centers 

No.  1 

No.  2 

No.  3 

No.  4 

MEC01 

Inter 

MEC04 

MEC26 

MEC03 

MEC02 

Inter 

MEC11 

MEC09 

MEC  03 

Frontier 

MEC03 

MEC04 

Frontier 

MEC04 

MEC05 

Inter 

MEC04 

MEC37 

MEC09 

MEC06 

Lower 

MEC03 

MEC  11 

MEC26 

MEC07 

Lower 

MEC04 

MEC37 

MEC  11 

MEC09 

MEC08 

Inter 

MEC  11 

MEC03 

MEC09 

Frontier 

MEC09 

MEC  10 

Lower 

MEC1 1 

MEC37 

MEC27 

MEC11 

Frontier 

MEC  11 

MEC  12 

Lower 

MEC  11 

MEC09 

MEC  13 

Inter 

MEC04 

MEC  11 

MEC09 

MEC37 

MEC  14 

Inter 

MEC04 

MEC  11 

MEC09 

MEC  15 

Lower 

MEC37 

MEC03 

MEC04 

MEC16 

Inter 

MEC  11 

MEC37 

MEC09 

MEC04 

MEC  17 

Inter 

MEC03 

MEC37 

MEC04 

MEC  18 

Upper 

MEC04 

MEC37 

MEC09 

MEC19 

Inter 

MEC11 

MEC03 

MEC37 

MEC04 

MEC20 

Lower 

MEC37 

MEC11 

MEC03 

MEC04 

MEC21 

Inter 

MEC1 1 

MEC03 

MEC37 

MEC22 

Lower 

MEC03 

MEC37 

MEC11 

MEC23 

Upper 

MEC04 

MEC09 

MEC  11 

MEC24 

Inter 

MEC37 

MEC04 

MEC  11 

MEC09 

MEC25 

Inter 

MEC1 1 

MEC37 

MEC04 

MEC09 

MEC26 

Frontier 

MEC26 

MEC27 

Frontier 

MEC27 

MEC28 

Inter 

MEC  11 

MEC03 

MEC26 

MEC29 

Inter 

MEC  11 

MEC03 

MEC30 

Inter 

MEC1 1 

MEC37 

MEC03 

MEC31 

Lower 

MEC11 

MEC03 

MEC32 

Inter 

MEC  11 

MEC26 

MEC03 

MEC33 

Upper 

MEC  11 

MEC03 

MEC37 

MEC34 

Inter 

MEC1 1 

MEC03 

MEC37 

MEC35 

Inter 

MEC03 

MEC37 

MEC36 

Inter 

MEC37 

MEC09 

MEC37 

Frontier 

MEC37 

MEC38 

Inter 

MEC04 

MEC  11 

MEC09 

MEC37 

MEC39 

Frontier 

MEC  39 

MEC40 

Upper 

MEC03 

MEC  11 

MEC37 

MEC41 

Lower 

MEC37 

MEC04 

MEC09 

MEC42 

Lower 

MEC03 

MEC  11 

MEC43 

Lower 

MEC04 

MEC  11 

MEC09 

MEC44 

Inter 

MEC  11 

MEC04 

MEC09 

MEC37 

MEC45 

Lower 

MEC  11 

MEC03 

MEC37 

MEC46 

Lower 

MEC11 

MEC03 

MEC37 

MEC47 

Inter 

MEC1 1 

MEC37 

MEC09 

MEC48 

Upper 

MEC  11 

MEC37 

MEC03 

MEC49 

Inter 

MEC1 1 

MEC09 

MEC  50 

Inter 

MEC37 

MEC03 

MEC11 

MEC04 

MEC51 

Inter 

MEC04 

MEC  11 

MEC09 

Four  MECs—  MEC03,  MEC1 1,  MEC37,  and  MEC39— are  on  the  frontier  for  two  of  the 
three  models.  For  three  of  the  four  MECs  just  mentioned,  the  type  of  model— staff  or 
financial— determines  if  they  are  on  the  frontier.  The  exception  is  MEC39.  MEC39  is  on 
the  frontier  for  Staff  Model  16  and  “Indexed"  Financial  Model  16  but  not  for  Financial 
Model  16.  In  this  case,  indexing  brings  MEC39  to  the  frontier.  It  is  important  to  note 
that  MEC03,  MEC1 1,  and  MEC37  are  on  the  frontier  only  for  Financial  Model  16  and 
“Indexed"  Financial  Model  16.  These  results  demonstrate  that  while  MEC03,  MEC1 1, 
MEC36,  and  MEC37  are  efficient,  their  efficiency  comes  at  the  expense  of 
“specialization."  Thus,  they  represent  improvement  opportunities  for  their  “associated" 
non-frontier  MECs,  while  at  the  same  time  can  improve  their  performance  in  another 
dimension  through  diversification. 

Comparisons  between  Table  4.1,  4.2,  and  4.3  are  also  helpful  in  identifying  MECs  which 
are  consistently  doing  a “good"  job  of  allocating  their  staff  and  financial  resources.  For 
example,  if  an  MEC  is  consistently  in  the  upper-quartile  range— either  as  a frontier  or  non- 
frontier center— they  may  merit  a close  examination.  Five  MECs  are  in  the  upper-quartile 
range,  but  not  always  as  a frontier  center,  for  all  three  models.  These  MEC's  are:  (1) 

MEC  1 1 ; (2)  MEC  18:  (3)  MEC23;  (4)  MEC39;  and  (5)  MEC40.  Although  these  MECs 
do  not  exhibit  consistent,  frontier-level  performance,  they  are  consistently  performing 
well  in  comparison  to  the  overall  set  of  MECs. 

4.2.  Cross-Model  Comparisons 

The  results  presented  in  Section  4.1  raise  two  sets  of  questions.  First,  why  are  certain 
MECs  on  the  frontier?  Is  it  due  to  a particular  output  or,  in  the  case  of  CMEPERID,  to 
one  of  its  components?  Is  it  due  to  the  level  of  the  MEC's  staff  and  financial  inputs  or  to 
one  of  each  input's  components?  The  response  to  this  set  of  questions  occupies  much  of 
this  section  and  Section  4.3. 

Second,  are  some  MECs  fundamentally  different?  Is  some  form  of  stratification  within  a 
particular  data  set  necessary?  The  answer  to  this  set  of  questions  is  more  complex.  It  is 
true  that  certain  “groups"  of  MECs  face  different  market  conditions.  Some  MECs 
operate  in  rural  areas,  others  in  urban  areas.  Some  MECs  are  newly  established,  others 
have  been  in  operation  for  years.  Information  and  analyses  presented  in  Sections  4.2  and 
4.3  provide  needed  insights  on  the  issues  just  raised  but  fall  short  of  a complete  response 
to  the  second  set  of  questions.  A partial  response  is  given  at  the  end  of  Section  4.3  and 
elaborated  upon  in  Section  5.3. 

Our  objective  in  performing  cross-model  comparisons— a structured,  step-by-step  analysis 
of  the  differential  effect  of  adding  and/or  disaggregating  an  output  and/or  disaggregating 
the  input— is  to  illustrate  DEA's  potential  for  understanding  what  the  data  reveal.  The 
cross-model  comparisons  are  illustrated  through  a series  of  tables.  The  salient  results 
from  each  table  are  summarized  and  analyzed  in  the  accompanying  text.  At  the 
conclusion  of  this  section,  two  “paths  to  improved  performance"  are  defined  and 
described  in  detail  for  a key  set  of  models. 
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Tables  4.4  through  4.6  summarize  the  key  results  for  each  set  of  models.  In  each  table, 
the  model  name  is  listed  under  the  first  column  heading.  Model.  The  identity  of  each 
MEC  in  the  upper-quartile  range  (i.e.,  the  top  25  percent  of  those  MECs  analyzed)  and  in 
the  lower-quartile  range  (i.e.,  the  bottom  25  percent  of  those  MECs  analyzed)  is  listed  in 
the  table.  Due  to  lack  of  space,  the  identity  of  the  MECs  in  the  inter- quartile  range  (i.e., 
the  middle  50  percent)  are  not  listed  in  Tables  4.4  through  4.6. 

The  upper-quartile  range  is  subdivided  into  the  frontier  MECs  and  the  non-frontier 
MECs.  Because  all  frontier  MECs  have  an  efficiency  of  1 .000,  they  are  listed  in 
sequential  order  for  each  model.  Non-frontier  MECs— both  those  in  the  upper-quartile 
range  and  those  in  the  lower-quartile  range— are  listed  in  descending  order  of  efficiency. 
To  determine  if  a particular  MEC  is  in  the  upper-quartile  range,  in  the  lower-quartile 
range,  or  in  the  inter-quartile  range  for  a given  model,  first  choose  the  MEC,  then  review 
the  appropriate  table.  The  complete  list  of  the  MECs  analyzed  are  listed  in  sequential 
order  in  the  tables  in  Section  4.1. 

Tables  4.4  and  4.5  summarize  the  key  results  for  each  of  the  16  staff  models  (see  Table 
4.4)  and  for  each  of  the  16  financial  models  (see  Table  4.5).  Table  4.6  summarizes  the 
key  results  for  each  of  the  16  “indexed"  financial  models.  These  results  will  be  examined 
in  detail.  This  approach  is  taken  as  a means  of  illustrating  the  step-by-step  progression 
referred  to  above. 

Consider  MEC36  and  MEC23.  Among  the  16  staff  models  shown  in  Table  4.4,  MEC36 
was  in  the  lower-quartile  range  once  (for  Staff  Model  2),  was  in  the  upper-quartile  range 
10  times  four  times  as  a frontier  center  (for  Staff  Models  8.  13,  14,  and  16)  and  six  times 
as  a non-frontier  center  (for  Staff  Models  4,  5,  7,  10,  11,  and  15),  and  was  in  the  inter- 
quartile range  five  times.  In  all  cases  where  MEC  36  was  on  the  frontier,  both 
FTEMOAVE  and  CMEPERID  were  disaggregated.  Four  of  the  six  cases  where  MEC36 
was  not  in  the  upper-quartile  range  were  associated  with  the  same  combination  of  outputs 
--ACT8  and  C PCTCME  (see  Table  3.3).  Among  the  16  staff  models,  MEC23  was  in 
the  upper-quartile  range  10  times  (for  Staff  Models  2,  3,  6.  7,  9,  11,  12,  14.  15,  and  16) 
and  was  in  the  inter-quartile  range  six  times.  In  all  cases  where  MEC23  was  in  the  upper- 
quartile  range,  the  same  output  was  present  — C PCTCME.  Thus  one  would  expect  that 
MEC23  has  achieved  good  market  penetration. 

Among  the  16  financial  models  (see  Table  4.5),  MEC36  was  in  the  lower-quartile  range 
twice  (for  Financial  Models  2 and  6)  and  in  the  inter-quartile  range  14  times.  Both  times 
MEC36  was  in  the  lower-quartile  range,  it  was  with  the  same  combination  of  outputs— 
ACT  8 and  C_PCTCME.  Among  the  16  financial  models,  MEC23  was  in  the  upper- 
quartile  range  1 1 times— each  time  not  as  a frontier  MEC— and  was  in  the  inter-quartile 
range  five  times. 

Among  the  16  “indexed"  financial  models  (see  Table  4.6),  MEC36  was  in  the  lower- 
quartile  twice  (for  “Indexed"  Financial  Models  2 and  6)  and  in  the  inter-quartile  range  14 
times.  Both  times  MEC36  was  in  the  lower-quartile  range,  it  was  with  the  same 
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combination  of  outputs— ACT  S and  C_PCTCME.  Thus  indexing  for  systematic 
differences  in  costs  faced  by  MEC36  is  not  sufficient  to  overcome  MEC36's  ‘'relatively” 
lower  levels  of  performance  on  ACT  8 and  C_PCTCME.  Among  the  16  financial 
models,  MEC23  was  in  the  upper-quartile  range  10  times— each  time  not  as  a frontier 
MEC— and  was  in  the  inter-quartile  range  six  times. 

Comparing  Table  4.6  to  Table  4.5  reveals  two  interesting  types  of  movement  for  MEC23. 
First,  MEC23  has  moved  out  of  the  upper-quartile  range  for  “Indexed"  Financial  Model 
8.  Examining  the  characteristics  of  “Indexed”  Financial  Model  8 reveals  that  it 
disaggregates  CMEPERID  into  its  two  constituent  parts  (see  Table  3.5).  These 
characteristics  are  also  present  in  Financial  Models  4,  10,  and  13  (see  Table  3.4)  and 
“Indexed"  Financial  Models  4,  10,  and  13  (see  Table  3.5)  for  which  MEC23  was  in  the 
inter-quartile  range.  Second,  in  all  other  cases,  MEC23’s  relative  position  within  the 
upper-quartile  range  has  improved.  Specifically,  for  “Indexed"  Financial  Models  12,  14, 
and  16  it  is  the  highest  ranked  non-frontier  MEC.  Examining  the  characteristics  of 
“Indexed"  Financial  Models  12,  14,  and  16  reveals  that  these  models  all  contain  the 
market  penetration  output  metric,  C_PCTCME,  and  the  disaggregated  financial  input 
metrics,  NDX  PER  and  NDXJTPS. 

Recall  that  our  objective  in  performing  cross-model  comparisons— a structured,  step-by- 
step  analysis  of  the  differential  effect  of  adding  and/or  disaggregating  an  output  and/or 
disaggregating  the  input— is  to  illustrate  DEA‘s  potential  for  understanding  what  the  data 
reveal.  Table  4.6  provides  a framework  for  conducting  another  aspect  of  cross-model 
comparisons.  To  illustrate  this  aspect,  consider  “Indexed"  Financial  Model  1.  For 
“Indexed"  Financial  Model  1,  there  are  two  basic  paths  for  analyzing  the  effects  just  cited 
(e.g.,  moving  from  the  two-output,  one  input  case,  “Indexed”  Financial  Model  1,  to  the 
four-output,  two-input  case,  “Indexed"  Financial  Model  16).  The  aim  of  the  “Path 
Analysis”  which  follows  is  to  gain  a better  understanding  of  what  causes  each  MEC  to 
move  to  the  frontier. 

Path  1 measures  the  differential  effects  by  first  adding  outputs,  then  disaggregating  one  of 
the  outputs,  CMEPERID,  into  its  component  parts,  and  finally  disaggregating  the  input 
NDX  TOT  into  its  component  parts.  Path  2 measures  the  differential  effects  by  first 
disaggregating  the  input  NDX_TOT  into  its  component  parts,  then  adding  outputs,  and 
finally  disaggregating  one  of  the  outputs,  CMEPERID,  into  its  component  parts.  Both 
Path  1 and  Path  2 start  with  “Indexed"  Financial  Model  1 and  end  with  “Indexed" 
Financial  Model  16. 

Taking  each  path  in  turn,  we  first  describe  the  key  characteristics  of  each  path.  We  then 
examine  the  composition  of  the  upper-  and  lower-quartile  ranges,  paying  particular 
attention  to  changes  (e.g.,  MECs  entering  the  frontier,  or  MECs  entering  or  leaving  either 
the  upper-  or  the  lower-quartile  ranges). 
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Table  4.  4 Summary  of  Key  Results  for  Models  Analyzed  with  DEA:  Based  on  ’’Standard”  Output  Metric  and  Staff  Input 

Metrics 
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Table  4.  5 Summary  of  Key  Results  for  Models  Analyzed  with  DEA:  Based  on  ’’Standard”  Output  Metrics  and  Financial 

Input  Metrics 


Table  4.  6 Summary  of  Key  Results  for  Models  Analyzed  with  DEA:  Based  on  "Standard”  Output  Metrics  and  "Indexed" 

Financial  Input  Metrics 
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Path  1 


Key  Characteristics 


Step  1:  “Indexed"  Financial  Model  1: 

The  basic  two  output,  ACT  8 and  CMEPERID, 
one  input,  NDXTOT,  case. 

Step  2:  “Indexed"  Financial  Model  9: 

Add  C_PCTCME  as  the  third  output;  maintain 
NDX  TOT  as  the  single  input. 

Step  3:  ‘"Indexed"  Financial  Model  15: 

Disaggregate  CMEPERID  into  its  two 
component  parts,  CMEFIRST  and  C CME  RP, 
thus  creating  a four  output  model;  maintain 

NDX  TOT  as  the  single  input. 

Step  4:  “Indexed"  Financial  Model  16: 

Disaggregate  NDX  TOT  into  its  two  component 
parts,  NDX  PER  and  NDX  OPS;  maintain  the 
four  outputs. 

Composition  of  Upper-Quartile  and  Lower-Quarti/e 


Step  1:  Frontier  MECs: 

Non-Frontier  MECs: 

MEC1 1,  MEC37 

MEC03,  MEC33,  MEC40,  MEC48. 

MEC27,  MEC39,  MEC19,  MEC09 

MEC30,  MEC32,  MEC25 

Lower-Quartile  MECs: 

MEC46,  MEC47,  MEC24,  MEC20, 

MEC17.  MEC05,  MEC12,  MEC10, 

MEC15,  MEC06,  MEC41.  MEC43,  MEC07 

Step  2:  Frontier  MECs: 

Non-Frontier  MECs: 

MEC04  is  added  to  the  frontier 

MEC23  and  MEC26  enter  the  upper- 
quartile  range,  MEC09,  MEC30,  and 

MEC32  move  to  the  inter-quartile  range, 
slight  change  in  rankings 

Lower-Quartile  MECs: 

MEC1 7 and  MEC05  move  to  the  inter- 
quartile range,  MEC22  and  MEC31  enter  the 
lower-quartile  range,  shift  in  rankings 
throughout  the  lower-quartile  range 
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Step  3:  Frontier  MECs: 

Non-Frontier  MECs: 

Lower-Quartile  MECs: 

Step  4:  Frontier  MECs: 

Non-Frontier  MECs: 

Lower-Quartile  MECs: 


MEC03  is  added  to  the  frontier 

No  change  in  membership,  slight  change  in 

rankings 

MEC47  moves  to  the  inter-quartile  range, 
MEC42  enters  the  lower-quartile  range,  shift 
in  rankings  throughout  the  lower-quartile 
range 

MEC09,  MEC26,  MEC27,  and  MEC39  are 
added  to  the  frontier 

MEC26,  MEC27,  and  MEC39  move  to  the 
frontier,  MEC18  enters  the  upper-quartile 
range,  MEC01  and  MEC19  move  to  the 
inter-quartile  range 

MEC43  moves  to  the  inter-quartile  range, 
MEC45  enters  the  lower-quartile  range,  shift 
in  rankings  throughout  the  lower-quartile 
range 


Path  2 


Key  Characteristics 

Step  1 : "Indexed"  Financial  Model  1 : The  basic  two  output,  ACT_8  and  CMEPERID, 

one  input,  NDX  TOT,  case. 

Step  2:  "Indexed"  Financial  Model  5:  Disaggregate  NDX_TOT  into  its  two  component 

parts,  NDX  PER  and  NDX  OPS;  maintain  the 
two  outputs. 

Step  3:  "Indexed"  Financial  Model  12:  Add  C_PCTCME  as  the  third  output;  maintain 

the  two  inputs. 

Step  4:  "Indexed"  Financial  Model  16:  Disaggregate  CMEPERID  into  its  two 

component  parts,  CMEFIRST  and  C CME  RP. 
thus  creating  a four  output  model;  maintain  the 
two  inputs. 


Composition  of  Upper-Quartile  and  Lower-Quartile 


Step  1: 


Step  2: 


Step  3: 


Step  4: 


Frontier  MECs: 
Non-Frontier  MECs: 


Lower-Quartile  MECs: 


MEC1 1,  MEC37 

MEC03,  MEC33,  MEC40,  MEC48, 

MEC27,  MEC39,  MEC  19,  MEC09, 

MEC30,  MEC32,  MEC25 
MEC46,  MEC47,  MEC24,  MEC20. 

MEC  17,  MEC05,  MEC12,  MEC10, 

MEC15,  MEC06,  MEC41,  MEC43,  MEC07 


Frontier  MECs: 
Non-Frontier  MECs: 


Lower-Quartile  MECs: 


MEC03  and  MEC09  are  added  to  the 
frontier 

MEC03  and  MEC09  move  to  the  frontier, 
MEC 1 8 enters  the  upper-quartile  range, 
MEC32  moves  to  the  inter-quartile  range, 
slight  change  in  rankings 
MEC24  moves  to  the  inter- 
quartile range,  MEC22  enters  the  lower- 
quartile  range,  shift  in  rankings  throughout 
the  lower-quartile  range 


Frontier  MECs: 
Non-Frontier  MECs: 


Lower-Quartile  MECs: 


MEC04  and  MEC26  are  added  to  the 
frontier 

MEC23  enters  the  upper- 
quartile  range,  MEC  19,  MEC25,  and 
MEC30  move  to  the  inter-quartile  range, 
shift  in  rankings  throughout  the  upper- 
quartile  range 
MEC05  moves  to  the  inter- 
quartile range,  MEC42  enters  the 
lower-quartile  range,  shift  in  rankings 
throughout  the  lower-quartile  range 


Frontier  MECs: 
Non-Frontier  MECs: 
Lower-Quartile  MECs: 


MEC27  and  MEC39  are  added  to  the 
frontier 

MEC27  and  MEC39  move  to  the  frontier, 

slight  change  in  rankings 

MEC  1 7 moves  to  the  inter-quartile  range, 

MEC45  enters  the  lower-quartile  range,  shift 

in  rankings  throughout  the  lower-quartile 

range 


34 


4.3.  Interpretation  of  Selected  Results 

The  previous  section  has  covered  a great  deal  of  material.  Consequently,  it  is  helpful  to 
pause  and  pull  together  our  findings.  We  begin  with  a synopsis  of  the  two  paths 
described  in  the  text  of  Section  4.2.  The  two  paths  are  summarized  graphically  in  Figure 
4. 1 . The  figure  provides  a schematic  diagram  of  a step-by-step  analysis  associated  with  a 
typical  cross-model  comparison  process.  The  figure  traces  each  step,  going  from  left  (i.e., 
the  Step  1,  two-output,  one-input  model)  to  right  (i.e.,  the  Step  4,  four-output,  two-input 
model).  The  figure  also  helps  us  to  interpret  the  results  of  our  DEA  analyses. 

In  interpreting  the  results  of  our  analyses,  we  shall  focus  on  those  MECs  which  defined 
the  production  frontier.  First,  we  see  that  two  MECs— MEC1 1 and  MEC37— are  on  the 
frontier  for  “Indexed"  Financial  Model  1.  Second,  there  are  eight  MECs  on  the  frontier 
for  “Indexed"  Financial  Model  16;  namely,  the  two  originals  plus  MEC03,  MEC04, 
MEC09,  MEC26,  MEC27,  and  MEC39.  Next,  we  examine  the  six  MECs  which  are 
ultimately  added  to  the  frontier.  Our  aim  is  to  gain  a better  understanding  of  what  is 
causing  them  to  be  frontier  centers.  This  understanding  can  be  used  to  encourage 
benchmarking  among  MECs  by  linking  specific  areas  of  excellence  for  a reference  center 
with  the  corresponding  opportunities  for  improvement  for  one  or  more  of  its  non-frontier 
MECs. 

We  begin  with  Path  2,  shown  in  the  lower  portion  of  Figure  4.1.  Two  MECs,  MEC03 
and  MEC09,  enter  the  frontier  when  NDX  TOT  is  disaggregated.  This  result  is  telling  us 
that  when  you  consider  the  financial  input  as  an  aggregated  total,  the  performance  of 
these  MECs  is  not  exemplary.  One  reason  why  these  MECs  might  move  to  the  frontier  is 
that  they  excel  in  the  allocation  of  their  financial  resources  among  staff  and  operating 
expenses.  If  this  is  the  case,  MEC03  and  MEC09,  represent  excellent  benchmarking 
opportunities  for  their  “associated"  non-frontier  MECs.  However,  a closer  examination 
of  the  data  revealed  that  this  “desired"  outcome  was  not  the  case.  The  two  MECs  in 
question  did  not  excel  at  “diversifying"  or  “balancing"  their  financial  resources  among 
staff  and  operating  expenses.  Rather  they  excelled  through  “specialization"  in  one  or  the 
other.  This  outcome  raises  a question  on  the  merits  of  disaggregating  the  financial  input. 
If  seeking  an  “ideal"  balance  is  desired,  then  disaggregating  the  financial  input  in  DEA 
analyses  only  helps  to  identify  a potential  set  of  “best  practice"  MECs.  Once  the  DEA 
analyses  have  been  performed  and  the  “paths"  analyzed,  it  becomes  necessary  to  review 
each  of  the  MECs  which  were  initially  on  the  frontier  (i.e.,  when  the  financial  input  was 
aggregated)  and  each  MEC  added  to  the  frontier  whenever  the  financial  input  is 
disaggregated.  Such  follow-up  analyses  could  include  scatter  plots,  or  similar  graphics- 
oriented  tools,  to  differentiate  cases  where  specialization  prevails  from  those  in  which 
diversification  prevails.  In  principle,  a template  could  be  designed  which  identifies  those 
cases  which  arise  out  of  “diversification"  and  out  of  “specialization." 
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2 Outputs, 

4 Outputs, 

1 Input 

Disaggregate 

2 Inputs 

Output 


CMEPERID 


Figure  4.  1 Measuring  the  Effects  of  Changing  the  Outputs  and  Inputs  on  the 

Production  Frontier 
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Two  MECs.  MEC04  and  MEC26,  are  added  to  the  frontier  when  the  output  C_PCTCME 
is  added.  Thus  one  could  conclude  that  these  MECs  are  doing  very  well  in  covering  their 
target  population  of  CMEs.  A review  of  the  operating  data  set  revealed  that  MEC04  and 
MEC26  were  in  the  upper-quartile  for  the  reported  coverage's  among  the  51  MECs 
analyzed.  It  is  worth  noting  that  the  two  MECs  with  the  highest  reported  coverage's, 
MEC19  and  MEC40,  were  not  added  to  the  frontier.  The  fact  that  the  two  MECs  with 
the  highest  reported  coverage's  are  not  on  the  frontier  highlights  an  important  strength  of 
DEA.  namely  that  it  provides  better  insights  into  relative  performance  whenever  multiple 
inputs  and  outputs  come  into  play. 

Next,  consider  the  two  MECs,  MEC27  and  MEC39.  These  MECs  exhibit  an  interesting 
pattern  of  performance.  For  Path  2,  these  MECs  enter  the  frontier  when  the  output 
CMEPERID  is  disaggregated;  for  Path  1,  they  enter  the  frontier  when  the  input 
NDXTOT  is  disaggregated.  To  interpret  this  outcome,  it  is  helpful  to  assess  whether 
one  effect  dominates  or  if  they  are  of  equal  importance.  We  shall  begin  by  referring  once 
again  to  Table  4.6.  The  table  reveals  that  MEC27  and  MEC39  are  on  the  frontier 
whenever  both  CMEPERID  a/?c/NDX_TOT  are  disaggregated  (i.e.,  “Indexed"  Financial 
Models  8,  13,  14,  and  16).  Therefore,  the  effects  are  of  equal  importance. 

Finally,  consider  MEC09  and  MEC26.  For  Path  1,  they  enter  the  frontier  when  the  input 
NDX  TOT  is  disaggregated.  For  Path  2,  MEC09  enters  the  frontier  when  NDX  TOT  is 
disaggregated,  while  MEC26  enters  the  frontier  when  the  output  metric  C PCTCME  is 
added.  Once  again  we  assess  whether  one  effect  dominates  or  if  they  are  of  equal 
importance.  Reference  to  Table  4.6  reveals  that  MEC09  is  on  the  frontier  whenever 
NDX  TOT  is  disaggregated,  implying  that  disaggregating  the  input  is  the  effect  which 
dominates.  In  the  case  of  MEC26,  it  is  on  the  frontier  only  when  three  characteristics  are 
present  simultaneously  in  the  model:  (1)  ACT  8;  (2)  CPCTCME;  and  (3)  NDX  TOT  is 
disaggregated.  These  characteristics  are  present  in  “Indexed"  Financial  Models  6.  12,  14, 
and  1 6,  implying  that  the  effects  are  of  equal  importance. 

Interpreting  the  results  of  DEA  analyses,  such  as  cross-model  comparisons,  helps  to 
sharpen  benchmarking  opportunities  among  frontier  and  non-frontier  MECs  (e.g.,  by 
focusing  on  those  effects  which  caused  a particular  MEC  to  move  to  the  frontier).  In 
addition,  the  interpretations  built  around  Figure  4.1  and  the  two  path  descriptions  given  in 
Section  4.2  raise  important  questions  which  point  to  other  areas  for  analysis.  These  areas 
include  whether  some  form  of  stratification  within  a particular  data  set  is  necessary. 

Consider  the  two  paths  which  formed  the  basis  for  our  cross-model  comparisons  of  the 
“indexed"  financial  models.  Both  paths  lead  to  the  same  types  of  questions  which  argue 
for  further  analysis;  namely,  the  shifts  in  rankings  due  to:  (1 ) disaggregating  CMEPERID 
into  its  two  component  parts;  and  (2)  disaggregating  NDX_TOT  into  its  two  component 
parts.  The  grouping  of  the  models  enables  us  to  examine  these  shifts  in  several  ways. 

For  Path  1.  we  can  examine  the  changes  between  Step  2 and  Step  3 for  the  first  case  and 
between  Step  3 and  Step  4 for  the  second  case.  For  Path  2,  we  can  examine  the  changes 
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between  Step  3 and  Step  4 for  the  first  case  and  between  Step  1 and  Step  2 for  the  second 
case.  In  the  first  case,  we  are  looking  for  how  the  relative  mix  between  first  time  and 
repeat  service  to  CMEs  affects  the  ranking.  Types  of  analyses  might  include  a scatter  plot 
of  the  raw  data  followed  by  a series  of  scaling  operations  (e.g.,  ratios  of  each  component 
to  the  whole),  with  follow-up  analyses  aimed  at  whether  the  MEC  was  new  or  well- 
established.  In  the  second  case,  we  are  looking  at  the  relative  efficiency  of  how 
(administrative  and  field)  staff  and  operations  resources  are  utilized.  In  addition  to  the 
types  of  analyses  suggested  for  the  previous  case,  evaluating  how  MECs  in  high-cost 
areas  compare  to  those  in  low-cost  areas  would  be  particularly  valuable.  Although  the 
“indexed"  financial  models  represent  an  important  step  in  this  direction,  the  results 
documented  in  Figure  4.1  demonstrate  that  probing  for  deeper  meaning  is  essential  if 
focused  benchmarking  activities  among  MEC  are  to  take  place.  Finally,  the  types  of 
analyses  just  described  will  provide  insights  on  the  pros  and  cons  of  data  stratification. 
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5.  Summary,  Conclusions,  and  Suggestions  for  Further  Research 

5.1.  Summary 

The  purpose  of  this  report  is  threefold.  First,  it  describes  how  DEA  could  be  applied  to 
assist  the  MEP  Program  and  its  network  of  MECs  measure  and  evaluate  their 
performance.  For  example,  the  following  key  concepts  are  described: 

• how  the  performance  frontier  enables  us  to  calculate  the  efficiency  of  a given  set  of 
MECs; 

• how  to  identify  MECs  which  can  serve  as  reference  centers  (i.e.,  suggest  paths  to 
improve)  for  other,  less  efficient  MECs;  and 

• how  to  use  index  measures  to  assess  the  success  of  the  overall  MEP  Program  in 
“pushing"  the  performance  frontier  outwards. 

Second,  it  uses  data  from  the  current  NIST/MEP  Management  Information  Reporting 
System  to  illustrate  ways  in  which  DEA  can  help  MEP  headquarters  and  its  network  of 
MECs  to  measure  their  performance.  Specific  results  from  DEA  analyses  of  this  MEC 
operating  data  set  include: 

• specification  of  a series  of  models  based  on  a unique  combination  of  key  inputs— staff 
and  financial  resources— and  outputs;  in  all.  48  models  were  evaluated; 

• specification  of  efficiency  ranges  and  identification  of  reference  centers  for  each  of 
the  three  “most-detailed"  models  evaluated; 

• identification  of  the  frontier  MECs,  non-frontier  MECs  in  the  upper-quartile  range, 
and  MECs  in  the  lower-quartile  range  for  each  of  48  models  evaluated:  and 

• cross-model  comparisons  illustrating  how  to  measure  the  differential  effects  of  adding 
a new  variable  or  disaggregating  an  existing  variable. 

Finally,  this  report  seeks  to  stimulate  feedback  and  discussion  among  those  engaged  in 
MEP  performance  assessment.  DEA  provides  a method  which  MEP  and  its  network  of 
MECs  can  use  jointly  to  make  sense  of  the  multi-dimensional  nature  of  MEC 
performance  data.  While  DEA  is  only  one  of  several  MEP  approaches  to  performance 
assessment,  it  is  a particularly  flexible  and  interactive  one.  With  this  report  MEP  is  in  a 
position  to  initiate  further  uses  of  DEA  which  will  prove  beneficial  to  both  the  MECs  and 
the  national  program. 
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5.2.  Conclusions 


This  research  was  motivated  by  the  MEP  Program's  twin  needs  to  both  measure 
performance  and  identify  specific  actions  to  improve  performance.  This  report 
demonstrates  how  DEA  meets  these  twin  needs.  In  addition,  the  material  presented  in 
this  report  leads  to  five,  fairly  specific,  conclusions  which  should  facilitate  the  use  of 
DEA  as  an  analytical  tool  for  the  MEP  Program. 

First,  the  feasibility  of  DEA  has  been  demonstrated  with  actual  MEP  data.  This 
conclusion  is  particularly  important  because  the  MEP  Program  has  an  extensive  set  of 
data  on  each  MEC.  In  particular,  the  Semi-Annual  Report,  which  is  submitted  by  each 
MEC  to  the  MEP  Program  headquarters  at  NIST,  provides  a wealth  of  information  for 
DEA  applications.  Consequently,  no  new  data  collection/data  development  effort  is 
required  in  order  to  implement  DEA  as  an  analytical  tool. 

Second.  DEA  permits  each  MEC 's  performance  to  be  measured  in  an  objective, 
consistent  manner.  Specifically,  each  MEC  is  evaluated  against  all  other  MECs  using  the 
same  set  of  input  and  output  variables.  Thus,  each  MEC’s  performance  is  measured 
relative  to  “actual  achieved’'  levels  of  performance  of  other  MECs,  rather  than  to  some 
theoretical  optimum.  In  addition,  DEA  is  sufficiently  flexible  to  adjust  for  systematic 
differences  among  MECs.  For  example,  cost  differences  between  MECs  operating  in 
high  cost  areas  and  those  operating  in  low  cost  areas  can  be  normalized  via  a cost  index. 
Similarly,  differences  between  newly-established  MECs  and  those  which  have  been  in 
existence  for  several  years  can  be  analyzed  by  comparing  the  relative  importance  of  first 
time  service  to  CMEs  versus  repeat  business.  Finally,  by  using  “personalized’'  weights  to 
uniquely  combine  each  MEC's  inputs  and  outputs  so  as  to  maximize  its  performance, 
DEA  presents  each  MEC  in  its  “most  favorable  light.” 

Third,  the  form  of  the  model  used  in  the  DEA  analyses  provides  insights  into  MEC 
performance . This  is  because  different  model  formulations  measure  different  types  of 
results.  MECs  which  are  on  the  frontier  for  one  model  formulation  are  not  necessarily  on 
the  frontier  for  another.  Similarly,  MECs  which  are  in  the  upper-quartile  range  for  one 
model  formulation  may  either  move  to  the  frontier,  stay  in  the  upper  quartile,  or  move  to 
another  quartile  for  a different  model.  Cross-model  comparisons  and  “path  analyses” 
employ  different  model  formulations  to  provide  insights  both  into  areas  of  relative 
strengths  and  opportunities  for  improvement.  In  particular,  cross-model  comparisons  and 
“path  analyses”  provide  the  means  for  identifying  which  MECs  have  moved  to  the 
frontier  and  why. 

Fourth,  DEA  produces  a w’ell-defined  set  of  reference  centers  for  each  non-frontier  MEC. 
Reference  centers  are  a core  concept  of  DEA,  since  they  provide  the  basis  for 
benchmarking  among  MECs.  These  “benchmarking  opportunities”  enable  specific  “areas 
of  excellence”  to  be  identified  and  used  to  drive  performance  improvement  for  non- 
frontier MECs.  Additional  work,  outlined  in  the  next  section,  will  provide  a framework 
for  stimulating  benchmarking  activities  among  MECs. 
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Fifth,  DEA  provides  a mechanism  for  measuring  the  performance  of  the  overall  MEP 
Program.  This  mechanism,  outlined  in  the  next  section,  shows  how  a time  series  of  DEA 
analyses  may  be  used  to  establish  a set  of  “rolling  benchmarks"  for  the  MEP  Program 
itself. 

5.3.  Suggestions  for  Further  Research 

As  the  MEP  Program  takes  steps  to  employ  DEA  as  a performance  improvement  tool,  it 
is  suggested  that  it  expand  its  efforts  in  the  following  areas.  These  areas  are  designed  to 
provide  the  MEP  Program  with  a suite  of  products  which  support  the  measurement  and 
evaluation  of  both  individual  MECs  and  the  overall  national  program's  efforts.  Four 
specific  products  are  proposed. 

5.3.1.  Selection  of  Variables  and  Models:  Towards  a “Standard”  Set  of  Variables 
and  Models 

The  results  presented  in  Chapter  4 demonstrate  how  a carefully  chosen  set  of  models  can 
be  used  to  examine  the  effects  that  introducing/disaggregating  outputs  and  inputs  have  on 
MEC  performance.  If  a “standard"  set  of  variables  and  models  were  developed,  it  would 
enable  MEP  to  evaluate  performance  on  at  least  two  fronts.  First,  it  enables  MEP 
headquarters  to  suggest  benchmarking  opportunities  between  MECs.  Second,  it  enables 
MEP  to  evaluate  its  success  in  pushing  the  performance  frontier  outward,  creating  a 
dynamic,  as  opposed  to  static,  improvement  process. 

A key  component  in  the  development  of  a standard  set  of  variables  and  models,  is  a 
critical  analysis  of  the  pros  and  cons  of  the  two  following  issues:  (1)  disaggregating  the 
staff  and  financial  inputs;  and  (2)  data  stratification.  Results  presented  in  Section  4.3 
showed  that  disaggregating  the  financial  input  can  produce  frontier  MECs  which 
specialize  in  one  of  the  two  financial  inputs  (i.e.,  either  staff  or  operating  expenses). 
Additional  research  and  analysis  is  needed  on  this  issue  to  determine  if  specialization  is  a 
“best  practice"  which  other  MECs  should  emulate.  Additional  research  is  needed  on  the 
data  stratification  issue  to  determine  if  some  alternative  grouping  of  MECs  would  yield 
results  which  lead  to  better  benchmarking  opportunities. 

As  the  MEP  Program  moves  toward  establishing  a standard  set  of  variables  and  models 
for  DEA  analyses,  it  should  work  to  capture  information  on  the  following  two  types  of 
variables.  First,  additional  work  is  needed  to  further  refine  and  apply  methods  of 
controlling  for  variations  in  costs  due  to  geographical  dispersion  among  MECs.  Second, 
priority  should  be  placed  on  expanding  the  capability  to  utilize  information  from 
additional  sources  (e.g.,  the  NIST/MEP  Short  Term  Follow-Up  Survey  Data)  as  DEA 
output  variables.  A key  advantage  of  this  approach  is  its  focus  on  “outcomes-oriented" 
metrics.  Another  advantage  is  the  potential  for  creating  linkages  between  REMI1" 

10  Treyz,  G.  I.,  Rickman.  D.  S.,  and  Shao,  G.  1992.  “The  REMI  Economic-Demographic  Forecasting  and 
Simulation  Model,"  International  Regional  Science  Review , Vol.  14,  pp.  221-253. 
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(Regional  Economic  Models,  Inc.)  and  DEA  analyses.  REMI  is  a tool  being  used  by  the 
Office  of  Applied  Economics  to  measure  the  economic  impacts  of  government 
technology  programs1 1 . 


5.3.2.  Feedback  Reports  for  Indiv  idual  Centers 

Once  a standard  set  of  DEA  variables  and  models  has  been  specified,  it  becomes  possible 
to  initiate  a more  focused  effort  at  continuous  improvement.  Feedback  reports  to  each 
MEC  are  one  means  for  focusing  an  MEC's  attention  on  both  its  strengths  and 
improvement  opportunities.  Ideally,  the  design  of  the  feedback  report  would  be  built 
around  a template  for  reporting  key  results.  The  MEP  Program  already  has  templates 
which  all  MECs  use  in  completing  their  Monthly  and  Semi-Annual  Reports.  While 
templates  are  widely  used  by  MECs,  work  is  needed  to  design  the  DEA  template  to  insure 
that  the  DEA-related  results  which  are  recorded  on  it  are  provided  in  a form  which  the 
MEC  can  easily  understand  and  act  upon.  The  feedback  report,  in  the  form  of  a template, 
thus  represents  an  important  step  towards  identifying  benchmarking  opportunities 
between  a non-frontier  MEC  and  its  reference  centers. 

5.3.3.  Process  for  Benchmarking  Among  Centers 

The  reference  center  concept  identifies  benchmarking  opportunities  for  non-frontier 
MECs.  In  order  to  ensure  that  the  benchmarking  opportunities  become  a “two-way 
street"  to  performance  improvement,  a process  for  benchmarking  among  MECs  needs  to 
be  developed  and  tested. 

Because  DEA  has  been  widely  used  in  geographically  distributed  settings  similar  to  those 
faced  by  MEP,  much  can  be  learned  from  the  literature  on  the  subject  and  through 
discussions  with  practitioners  and  veterans  in  the  field.  Our  goal  is  to  produce  a process 
which  is  clear  and  simple  but  robust  enough  to  produce  results. 

5.3.4.  Process  for  Measuring  the  Performance  of  the  MEP  Program 

If  a standard  set  of  variables  and  models  is  established,  then  an  important  aspect  of 
overall  MEP  Program  performance  can  be  measured  through  an  index.  The  index  uses  a 
time  series  of  the  optimally-calculated  weights  from  DEA  (i.e.,  a set  of  weights  for  each 
frontier  center  at  different  points  in  the  time  series)  to  determine  if  the  performance 
frontier  is  being  pushed  outward.  From  the  overall  MEP  Program  perspective,  pushing 
the  performance  frontier  outward  is  a desired  outcome  because  it  implies  greater 
“absolute"  efficiency. 


11  Ehlen,  Mark  A.,  and  Weber,  Stephen  F.  1997.  Estimating  Economic  Impacts  of  Government  Technology 
Programs:  Manufacturing  Studies  Using  the  REMI  Model.  NISTIR  6107.  Gaithersburg,  MD:  National 
Institute  of  Standards  and  Technology. 
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Appendix  A.  A Graphical  Approach  to  Performance  Assessment 

The  purpose  of  this  appendix  is  to  present  a graphical  approach  to  performance 
assessment  which  illustrates  two  key  DEA  concepts.  First,  the  appendix  describes  how 
the  efficiency  of  a non-frontier  MEC  is  calculated.  Second,  using  a graphical 
representation  of  efficiency,  a set  of  optimal  weights  is  derived.  Data  on  three  MECs  are 
used  to  illustrate  how  the  graphical  approach  would  be  applied  in  practice. 

Consider  the  case  of  Staff  Model  1 . This  two  output,  one-input  model  had  ACT_8  and 
CMEPERID  as  outputs  and  FTETOTAL  as  the  input.  For  our  purposes  here,  we  shall 
denote  ACT  8 as  output  1,  Oi,  and  CMEPERID  as  output  2,  02,  respectively.  We  denote 
the  single  input.  FTETOTAL  as  I. 

Figure  A.l  plots  the  results  of  Staff  Model  1 for  three  MECs.  C002.  C003,  and  COOT 
Two  of  these  MECs.  C002  and  COOS,  are  frontier  MECs.  The  third  MEC.  COOL  is  a non- 
frontier MEC.  In  developing  Figure  A.l,  the  ratio  Oi  of  over  I.  is  designated  as  X and 
the  ratio  of  O2  over  I is  designated  as  Y.  This  allows  us  to  use  the  convenient  geometric 
representation  of  the  XY-axis.  where  X is  the  horizontal  axis  and  Y is  the  vertical  axis. 
The  values  used  to  construct  Figure  A.l  are  summarized  in  Table  A.l. 

Table  A.  1 Summary  of  Key  MEC  Operating  Data 


MEC 

ACT_8 

CMEPERID 

FTETOTAL 

Oi/I 

02/I 

ID 

Oi 

02 

I 

X 

Y 

C001 

1191 

46 

7.2S91 

25.8913 

C002 

188 

206 

13 

14.4615 

15.8462 

COOS 

47 

468 

9 

5.2222 

52.0000 

Refer  now  to  Figure  A.l.  For  COOL  plotted  as  point  A on  the  figure,  both  COOS,  plotted 
as  point  E,  and  C002.  plotted  as  point  F.  serve  as  reference  centers.  In  addition,  we  can 
identify  a target  point.  B.  on  the  frontier  which  represents  the  C001  MECs  potential.  In 
the  case  of  the  C001  MEC.  the  target  point  is  B on  the  frontier  and  COOEs  efficiency  is 
given  by  the  ratio  OA/OB.  Simple  geometry  establishes  that  the  ratio  OA/OB  is  the  same 
as  the  ratio  OC/OD.  The  point  D is  given  by  the  line  BD  obtained  by  extending  the  line 
joining  COOS  and  C002  to  the  Y-axis.  The  point  C is  given  by  the  intercept  on  the  Y-axis 
of  the  line  AC  drawn  parallel  to  BD  through  the  point  A (i.e..  C001). 
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Figure  A.l  Efficiency  Calculations  for  the  C001  MEC 
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The  equation  of  the  line  BD  is 


Y = 72.4348  - 3.913  *X 


Eq.A.l 


where  Y is  CMEPERID/FTETOTAL  (i.e.,  02/I),  X is  ACT  8/FTETOTAL  (i.e.,  0,/I), 

IT 

-3.913  is  the  slope  of  the  line,  ~ and  72.4348  is  the  value  of  the  intercept  of  the  line  on  the 
Y-axis  (OD).  The  equation  of  the  line  AC  is 

Y = 54.2183  - 3.913  * X Eq.A.2 

where  54.2183  is  the  value  of  the  intercept  of  the  line  on  the  Y-axis  (OC).  Hence,  our 
measure  of  COOEs  efficiency  is  given  by 


04 

OB 


OC 

~OD 


54.2183 

72.4348 


0.7485 


Eq.A.3 


Let  us  explore  equations  A.l  and  A. 2 further.  For  the  purpose  of  generalization,  let  us 
denote  the  coordinates  of  the  three  MECs,  C001,  C002,  and  C003,  as  (X(C001), 
Y(C001)),  (X(C002),  Y(C002)),  and  (X(C003),  Y(C003)),  respectively.  Then,  by 
substitution  we  have 


OC  - 54.2183  = Y(C001)  + 3.913  * 

OD  = 72.4348  = Y(C002)  + 3.913  * 

OD  - 72.4348  = Y(C003)  + 3.913  * 


X(C001) 

Eq.A.4 

X(C002) 

Eq.A.5 

X(C003) 

Eq.A.6 

Dividing  all  three  equations  by  OD  (=72.4348)  and  switching  the  left  and  right  hand  sides 
of  the  equations  gives  us 

0.0138  * Y(C001)  + 0.054  * X(C001)  = 0.7485  Eq.A.7 

0.0138  * Y(C002)  + 0.054  * X(C002)  = 1 Eq.A.8 


12  The  slope  of  the  line  is  equal  to  [ (Y(C003)  - Y(C002))  / (X(C003)  - (X(C002)) ) or,  in  numeric  terms, 
(52.0000  - 15.8462)7(5.2222  - 14.4615). 
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0.0138  * Y(C003)  + 0.054  * X(C003)  = 1 


Eq.A.9 


We  originally  obtained  the  X and  Y values  by  dividing  each  MEC's  outputs  by  its  input. 
Let  us  define  Oi(COOl)  and  02(0001)  to  be  the  C001  MEC's  outputs  and  I(C001)  to  be 
its  input  and  assume  similar  notation  for  the  frontier  MECs,  C002  and  0003,  respectively. 
Hence  we  have 


Y (C001)  = 


02(C001) 

I(C001) 


Eq.A.10 


X(C001) 


0,(0001) 

I(C001) 


Eq.A.ll 


Similar  relationships  exist  for  the  C002  and  C003  MECs.  Substituting  Eq.  A.  10  and 
A.l  1 into  Eq.  A. 7 through  A. 9 gives 


Q.0 1 38  *02(C001)  + 0.054*0,(0001) 
I(C001) 


0.7485 


Eq.A.12 


0.0138 *02(C002)  + 0.054  * 0,(0002) 

1(0002) 


Eq.A.13 


0.0138 *02(C003)  + 0.054  * 0,(0003) 
1(0003) 


Eq.A.14 


Finally,  we  can  constrain  the  denominator  of  equation  A.  12  to  be  equal  to  a value  of  1 by 
dividing  both  the  numerator  and  denominator  of  equation  A.  12  by  the  value  of  1(0001 ) 
(i.e.,  46,  see  Table  A.l).  We  repeat  the  same  step  (i.e.,  dividing  the  numerator  and 
denominator  by  46)  for  equations  A.  1 3 and  A.  14.  This  gives 
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0.0003*  Ch(COOl)  + 0.0017  * Oi(COOl) 

= 0.7485 

0.02174  * I(C001) 


Eq.A.15 


0.0003 *02(C002)  + 0.00117  * Qi(C002) 
0.02174  * I(C002) 


Eq.A.16 


0.0003  *P2(C003)  + 0.00117  * Qi(C003) 
0.02174  * I(C003) 


Eq.A.l  7 


The  purpose  of  this  constraint  is  to  accord  with  the  solution  procedure  outlined  in  Section 
2.3. 

Thus,  we  have  expressed  our  efficiency  measure  for  the  C001  MEC  (0.7485)  as  the  ratio 
of  the  weighted  sum  of  its  outputs  to  the  weighted  sum  of  its  input  while  restricting  the 
value  of  the  weighted  sum  of  its  input  to  1 . Furthermore,  the  value  of  the  weighted  sum 
ratio  obtained  using  the  same  weights  for  either  of  COOEs  reference  MECs,  C002  and 
C003,  is  1 (refer  to  equations  A.  16  and  A.  17). 

Also,  since  all  of  the  inputs  and  outputs  have  positive  values,  it  is  clear  by  direct 
inspection  of  the  ratios  (i.e.,  equations  A.  15,  A.  16,  and  A.  17)  that,  if  we  increase  either  of 
the  output  weights  or  decrease  the  input  weight  in  order  to  increase  the  computed 
efficiency  for  C001,  then  an  identical  change  to  either  the  C002  or  C003  ratio  will  cause 
it  to  exceed  the  value  1 . Hence,  the  weights  obtained  maximize  the  value  of  a weighted 
sum  ratio,  subject  to  the  constraint  that  the  same  weighted  ratio  of  any  other  MEC's 
outputs  to  inputs  must  not  exceed  the  value  1 . 

In  summary,  we  have  demonstrated  that  our  graphical  efficiency  measure  for  the  C001 
MEC  obtained  from  Figure  A.l  is  equivalent  to  the  value  of  the  ratio  of  a weighted  sum 
of  its  outputs  and  a weighted  sum  of  its  input,  M’here  the  weights  are  selected  to  maximize 
the  value  of  the  ratio  subject  to  the  value  of  the  same  weighted  ratio  of  any  other  MEC's 
outputs  and  inputs  not  exceeding  1 . Let  us  now  consider  further  how  the  weights  should 
be  interpreted.  If  we  return  to  equations  A. 4,  A. 5 and  A. 6,  we  see  that  the  coefficients  of 
Y and  X in  each  case  are  1 and  3.91 3,  where  the  latter  is  the  negative  of  the  slope  of  the 
line  BD.  In  the  subsequent  development  to  generate  the  ratio  of  weighted  sums,  the 
output  weights  were  maintained  in  this  proportion.  In  other  words,  the  ratio  of  the  output 
weights  is  equal  to  the  slope  of  the  line  on  the  frontier  which  contains  the  point  against 
which  efficiency  is  measured.  This  slope  represents  the  trade-off  between  the  two  outputs 
at  that  point. 
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Note  that  any  other  MEC  within  the  shaded  triangle  OEF  would  also  be  measured  with 
respect  to  a point  on  the  line  segment  EF.  Hence  a further  consequence  of  the  above 
result  is  that  all  MECs  in  the  triangle  have  output  weights  in  the  same  proportion. 

Finally,  if  there  were  another  MEC  within  the  triangle  OEF  and  also  on  the  line  AC,  the 
equations  for  this  MEC  would  be  identical  to  those  for  C001,  resulting  in  an  identical 
efficiency  score  being  achieved.  Hence  efficiency  contours  exist  and  are  parallel  to  the 
frontier. 

It  is  critical  to  recognize  that  the  weights  represent  the  trade-offs  which  apply  across  a 
frontier  segment.  However,  by  virtue  of  the  fact  that  these  weights  maximize  the 
weighted  sum  ratio  of  outputs  to  inputs  within  the  constraints  imposed,  we  can  be  certain 
that  no  other  MEC  will  ever  be  presented  in  a better  light  by  an  alternative  method  of 
weighting.  Equivalently,  we  can  be  certain  that  DEA  will  never  overestimate 
improvement  potential  relative  to  that  identified  by  any  other  weighting  method.  Thus, 
the  weights  for  each  MEC  are  uniquely  determined  so  as  to  maximize  its  efficiency 
subject  to  the  constraint  that  no  other  MEC  can  have  an  efficiency  greater  than  1 with  the 
same  weights. 
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Appendix  B. 


Mathematical  Formulation 


Consider  the  one-input,  two-output,  problem  discussed  in  Appendix  A.  The  efficiency  of 
each  MEC  on  the  frontier  was  defined  as  being  equal  to  1 .0.  Additionally,  the  efficiency 
of  the  MEC  not  on  the  frontier  was  calculated  as  the  ratio  of  two  radial  distances  defined 
by  its  plotted  performance. 

Norman  and  Stoker  have  shown  that  this  graphical  approach  to  calculating  efficiencies  is 
equivalent  to  calculating  efficiencies  as  the  ratio  of  the  weighted  sum  of  the  MEC's 
outputs  divided  by  the  weighted  sum  of  its  inputs,  where  each  MEC's  weights  must 
satisfy  two  sets  of  constraints.  First,  the  weights  are  all  constrained  to  be  non-negative. 
Second,  the  weights  for  each  MEC  are  chosen  to  maximize  its  calculated  efficiency, 
subject  to  the  restriction  that  neither  its  nor  any  other  MEC's  efficiency  using  these  same 
weights  exceed  1 .0. 


This  problem  statement  can  be  generalized  to  the  case  with  O outputs  and  / inputs. 
Denoting  the  inputs  and  outputs  of  the  Ath  MEC  as 

inputs:  jc  , i = 1, ....  / 

outputs:  yjkJ=  U •••>  O 

the  problem  can  be  stated  as  follows,  where  A is  the  number  of  MECs  included  in  the 
analysis: 


subject  to: 


m = 1,  ...,  N 


w >0;  /=1, ...,  O 

J u 


v,  > 0: 


7=1 


The  definition  above  is  not  a linear  programming  problem  because  the  objective  function 
is  not  a linear  function  of  the  inputs.  In  their  landmark  paper  which  introduced  the  DEA 
method.  Chames,  Cooper  and  Rhodes1'1  introduced  the  following  formulation  as  a re- 
statement of  this  problem  in  linear  programming  terms: 


1 ’ Chames,  A.,  Cooper.  W.,  and  Rhodes.  E.  1978.  “Measuring  the  Efficiency  of  Decision  Making  Units." 
European  Journal  of  Operational  Research.  Vol.  2.  pp.  429-444. 
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max^=X'li  w,y» 


subject  to: 

Z%x»»  - Z°  i - 0;  m = 1 - •••• N 
X,=1  vixik  = 1 

w,  SO;  / = 1, ....  0 
v,  >0;  /=  1,  ....  1 


When  solving  the  linear  programming  problem  for  the  Ath  MEC,  the  input  weights  are 
selected  so  that  the  weighted  sum  of  the  inputs  equals  1 .0.  The  output  weights  are 
selected  to  maximize  the  weighted  sum  of  the  outputs,  and  thus  the  efficiency  of  the  Ath 
MEC.  Thus,  weights  are  selected  which  show  each  MEC  in  the  best  possible  light.  Does 
this  mean  that  all  MECs  are  found  to  be  100%  efficient?  No,  because  the  first  of  the  four 
restriction  equations  listed  above  requires  that  no  other  MEC  be  found  more  than  100% 
efficient  using  those  same  input  and  output  weights  with  its  own  levels  of  inputs  and 
outputs. 

Note  that  this  linear  programming  problem  is  solved  for  the  Ath  MEC.  Thus,  n of  these 
separate  linear  programming  problems  must  be  solved  to  generate  the  full  set  of 
efficiency  scores. 

Any  linear  programming  problem  has  an  equivalent  “dual"  linear  programming  problem 
whose  solution  is  mathematically  equivalent  to  the  original  or  “primal"  formulation.14 
The  dual  formulation  is  crucial  in  the  application  of  DEA,  since  it  is  the  vehicle  through 
which  the  Ath  MEC's  reference  centers  are  identified.  The  identification  of  reference 
centers  is  accomplished  through  a post-processing  operation  which  examines  the  values 
of  the  dual  variables  associated  with  each  constraint  in  the  original  or  primal  formulation. 
Non-zero  values  of  the  dual  variables  imply  that  the  constraint  is  binding  (i.e.,  the 
efficiency  of  the  Ath  MEC  would  be  higher  if  that  constraint  were  relaxed).  Those 
constraints  which  are  binding  for  the  Ath  MEC  taken  together  define  a “facet"  on  the 
performance  frontier.  The  vertices  of  the  facet  correspond  to  the  Ath  MEC’s  reference 
centers.  Thus,  to  form  the  set  of  reference  centers  for  the  Ath  MEC,  it  is  sufficient  to 
examine  the  WMEC-related  constraints  and  identify  which  MECs  are  associated  with  the 
constraints  which  are  binding  (i.e.,  whose  dual  variables  have  non-zero  values). 


14  See,  for  example,  Gass,  Saul  I.  1985.  Linear  Programming.  Fifth  Edition.  New  York:  McGraw-Hill 
Book  Company 
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